A simple library for creating complex neural networks
java.lang.Error when using Executors.singleThreadExecutor as the execution context.Port all built-in plugins except INDArray-related plugins to Scala 2.12.
Thanks to @lrytz (https://github.com/scala/bug/issues/10334, https://github.com/scala/scala/pull/5973 and https://github.com/scala/scala/pull/5977).
Today, we are happy to announce DeepLearning.scala 2.0.0, the new stable release of DeepLearning.scala, a simple library for creating complex neural networks from object-oriented and functional programming constructs.
map, reduce or other higher order functions.In DeepLearning.scala 2.0, we removed the special support for differentiable ADT and Boolean types. Now differentiable computational graphs are ordinary Scala code, so all types including ADT and Boolean are avialable in these graphs.
Unlike some other deep learning frameworks, the structure of neural networks in DeepLearning.scala is dynamically determined during running. Our neural networks are programs. All Scala features, including functions and expressions, are available in neural networks.
For example:
def ordinaryScalaFunction(a: INDArray): Boolean = {
a.signnum.sumT > math.random
}
def myDynamicNeuralNetwork(input: INDArray) = INDArrayLayer(monadic[Do] {
val outputOfLayer1 = layer1(input).forward.each
if (ordinaryScalaFunction(outputOfLayer1.data)) {
dynamicallySelectedLayer2(outputOfLayer1).forward.each
} else {
dynamicallySelectedLayer3(outputOfLayer1).forward.each
}
})
The above neural network will go into different subnetworks according to an ordinary Scala function.
With the ability of creating dynamic neural networks, regular programmers are able to build complex neural networks from simple code. You write code almost as usual, the only difference being that code based on DeepLearning.scala is differentiable, which enables such code to evolve by modifying its parameters continuously.
DeepLearning.scala 2.0 is based on Monads, which are composable, thus a complex layer can be built from primitive operators. Along with the Monad, we provide an Applicative type class, to perform multiple calculations in parallel.
For example, the previous example can be rewritten in higher-order function style as following:
def myDynamicNeuralNetwork(input: INDArray) = INDArrayLayer {
layer1(input).forward.flatMap { outputOfLayer1 =>
if (ordinaryScalaFunction(outputOfLayer1.data)) {
dynamicallySelectedLayer2(outputOfLayer1).forward
} else {
dynamicallySelectedLayer3(outputOfLayer1).forward
}
}
}
The key construct in DeepLearning.scala 2.0 is the dependent type class DeepLearning, which witnesses a differentiable expression. In other words, given the DeepLearning type class instance, you can activate the deep learning ability of any type.
The code base of DeepLearning.scala 2.0 is organized according to Dependent Object Type calculus (DOT). All features are provided as mixin-able plugins. A plugin is able to change APIs and behaviors of all DeepLearning.scala types. This approach not only resolves expression problem, but also gives plugins the additional ability of virtually depending on other plugins.
For example, when a plugin author is creating the Adagrad optimizer plugin, he does not have to explicitly call functions related to learning rate. However, once a plugin user enables both the Adagrad plugin and the FixedLearningRate plugin, then computation in FixedLearningRate will get called eventually when the Adagrad optimization is executed.
| Plugin Name | Plugin Description |
|---|---|
| Builtins | All the built-in plugins. |
| FixedLearningRate | Setup fixed learning rate when training INDArray weights. |
| Adagrad | An adaptive gradient algorithm with per-parameter learning rate for INDArray weights. |
| L1Regularization | L1 Regularization. |
| L2Regularization | L2 Regularization. |
| Momentum | The Momentum and NesterovMomentum optimizer for SGD. |
| RMSprop | The RMSprop optimizer for SGD. |
| Adam | The Adam optimizer for SGD. |
| INDArrayDumping | A plugin to dump weight matrices during training. |
| CNN | A standalone CNN implementation. |
| Add your own algorithms, models or any cool features here. | |
DeepLearning.scala 2.0.0-RC1 is a release candidate of DeepLearning.scala 2.
DeepLearning.scala 2.0 comes with two major features in addition to DeepLearning.scala 1.0: dynamic neural network and Factory-based plugins.
In DeepLearning.scala 2.0, a neural network is an ordinary Scala function that returns a Layer, which represents the process that dynamically creates computational graph nodes, instead of static computational graphs in TensorFlow or some other deep learning frameworks. All Scala features, including functions and expressions, are available in DeepLearning.scala's dynamic neural networks.
Factory-based plugins resolve expression problem. Any hyperparameters, neural network optimization algorithms or special subnetworks are reusable in the simple Factory[YouPlugin1 with YouPlugin2] mechanism.
See Getting Started to have a try.
Version 1.0.0 is the first stable release.
Like Theano and other deep learning toolkits, DeepLearning.scala allows you to build neural networks from mathematical formulas. It supports floats, doubles, GPU-accelerated N-dimensional arrays, and calculates derivatives of the weights in the formulas.
Neural networks created by DeepLearning.scala support ADT data structures (e.g. HList and Coproduct), and calculate derivatives through these data structures.
Neural networks created by DeepLearning.scala may contains control flows like if/else/match/case in a regular language. Combined with ADT data structures, you can implement arbitary algorithms inside neural networks, and still keep some of the variables used in the algorithms differentiable and trainable.
Neural networks created by DeepLearning.scala are composable. You can create large networks by combining smaller networks. If two larger networks share some sub-networks, the weights in shared sub-networks trained with one network affect the other network.
All of the above features are statically type checked.
DeepLearning.scala is heavily inspired by @MarisaKirisame. Originally, we worked together for a prototype of deep learning framework, then we split our work aprt to this project and DeepDarkFantasy.
@milessabin's shapeless provides a solid foundation for type-level programming as used in DeepLearning.scala.