Building Annoy Index on Apache Spark
Building Annoy Index on Apache Spark. Then query neighbors using Annoy.
I had built an index of 117M 64-dimensional vectors using 100 nodes in 5 minutes. The settings was;
// version: 0.1.4
// spark.executor.instances = 100
// spark.executor.memory = 8g
// spark.driver.memory = 8g
val fraction = 0.00086 // for about 100k samples
val numTrees = 2
val numPartitions = 100
val annoyModel = new Annoy().setFraction(fraction).setNumTrees(numTrees).fit(dataset)
annoyModel.saveAsAnnoyBinary("/hdfs/path/to/index", numPartitions)
The size of the index is about 33G.
import spark.implicits._
val data = spark.read.textFile("data/annoy/sample-glove-25-angular.txt")
.map { str =>
val Array(id, features) = str.split("\t")
(id.toInt, features.split(",").map(_.toFloat))
}
.toDF("id", "features")
val ann = new Annoy()
.setNumTrees(2)
val annModel = ann.fit(data)
annModel.saveAsAnnoyBinary("/path/to/dump/annoy-binary")
From the version 0.1.2, it is released to Maven.
libraryDependencies += "com.github.mskimm" %% "ann4s" % "0.1.5"
0.1.5
is built with Apache Spark 2.3.0...
val training: DataFrame = _
val als = new ALS()
.setMaxIter(5)
.setRegParam(0.01)
.setUserCol("userId")
.setItemCol("movieId")
.setRatingCol("rating")
val model = als.fit(training)
val ann = new Annoy()
.setNumTrees(2)
.setFraction(0.1)
.setIdCol("id")
.setFeaturesCol("features")
val userAnnModel= ann.fit(model.userFactors)
userAnnModel.writeAnnoyBinary("exp/als/user_factors.ann")
val itemAnnModel = ann.fit(model.itemFactors)
itemAnnModel.writeAnnoyBinary("exp/als/item_factors.ann")
...
I personally started this project to study Scala. I found out that Annoy is a fairly good library for nearest neighbors search and can be implemented distributed version using Apache Spark. Recently, various bindings and implementations have been actively developed. In particular, the purpose and usability of this project overlap with some projects like annoy4s and annoy-java in terms of running on JVM.
To continue contribution, from now on this project focuses on building Index on Apache Spark for distributed builds. This will support building using 1 billion or more items and writing Annoy compatible binary.