Run inference on replit-3B code instruct model using CPU
Run inference on the replit code instruct model using your CPU. This inference code uses a ggml quantized model. To run the model we'll use a library called ctransformers that has bindings to ggml in python.
Demo:
Using docker should make all of this easier for you. Minimum specs, system with 8GB of ram. Recommend to use python 3.10
.
Will post some numbers for these two later.
First create a venv.
python -m venv env && source env/bin/activate
Next install dependencies.
pip install -r requirements.txt
Next download the quantized model weights (about 1.5GB).
python download_model.py
Ready to rock, run inference.
python inference.py
Next modify inference script prompt and generation parameters.