Believe in AI democratization. llama for nodejs backed by llama-rs, llama.cpp and rwkv.cpp, work locally on your laptop CPU. support llama/alpaca/gpt4all/vicuna/rwkv model.
llama-node: Node.js Library for Large Language Model
Picture generated by stable diffusion.
This project is in an early stage and is not production ready, we do not follow the semantic versioning. The API for nodejs may change in the future, use it with caution.
This is a nodejs library for inferencing llama, rwkv or llama derived models. It was built on top of llm (originally llama-rs), llama.cpp and rwkv.cpp. It uses napi-rs for channel messages between node.js and llama thread.
llama.cpp backend supported models (in GGML format):
llm(llama-rs) backend supported models (in GGML format):
rwkv.cpp backend supported models (in GGML format):
Node.js version: >= 16
npm install llama-node
Install anyone of the inference backends (at least one)
npm install @llama-node/llama-cpp
npm install @llama-node/core
npm install @llama-node/rwkv-cpp
Please see how to start with manual compilation on our contribution guide
Please read the document on our site to get started with manual compilation related to CUDA support
This library was published under MIT/Apache-2.0 license. However, we strongly recommend you to cite our work/our dependencies work if you wish to reuse the code from this library.
Join our Discord community now! Click to join llama-node Discord