A universal scalable machine learning model deployment solution
DJL Serving is a high performance universal stand-alone model serving solution powered by DJL. It takes a deep learning model, several models, or workflows and makes them available through an HTTP endpoint. It can serve the following model types out of the box:
You can install extra extensions to enable the following models:
For macOS
brew install djl-serving
# Start djl-serving as service:
brew services start djl-serving
# Stop djl-serving service
brew services stop djl-serving
For Ubuntu
curl -O https://publish.djl.ai/djl-serving/djl-serving_0.27.0-1_all.deb
sudo dpkg -i djl-serving_0.27.0-1_all.deb
For Windows
We are considering to create a chocolatey
package for Windows. For the time being, you can
download djl-serving zip file from here.
curl -O https://publish.djl.ai/djl-serving/serving-0.27.0.zip
unzip serving-0.27.0.zip
# start djl-serving
serving-0.27.0\bin\serving.bat
You can also use docker to run DJL Serving:
docker run -itd -p 8080:8080 deepjavalibrary/djl-serving
DJL Serving can be started from the command line. To see examples, see the starting page.
djl-serving --help
usage: djl-serving [OPTIONS]
-f,--config-file <CONFIG-FILE> Path to the configuration properties file.
-h,--help Print this help.
-m,--models <MODELS> Models to be loaded at startup.
-s,--model-store <MODELS-STORE> Model store location where models can be loaded.
-w,--workflows <WORKFLOWS> Workflows to be loaded at startup.
See configuration for more details about defining models, model-store, and workflows.
DJL Serving uses a RESTful API for both inference and management calls.
When DJL Serving starts up, it has two web services:
By default, DJL Serving listens on port 8080 and is only accessible from localhost. Please see DJL Serving Configuration for how to enable access from a remote host.
Details about how DJL Serving is implemented can be found in the architecture docs.
DJL Serving supports plugins, user can implement their own plugins to enrich DJL Serving features. See DJL Plugin Management for how to install plugins to DJL Serving.
you can set the logging level on the command-line adding a parameter for the JVM
-Dai.djl.logging.level={FATAL|ERROR|WARN|INFO|DEBUG|TRACE}