Vector search demo with the arXiv paper dataset, RedisVL, HuggingFace, OpenAI, Cohere, FastAPI, React, and Redis.
This repository is the official codebase for the arxiv paper search app hosted at: https://docsearch.redisvl.com
Redis is a highly performant, production-ready vector database, which can be used for many types of applications. Here we showcase Redis vector search applied to a document retrieval use case. Read more about AI-powered search in the technical blog post published by our partners, Data Science Dojo.
The arXiv papers dataset was sourced from the the following Kaggle link. arXiv is commonly used for scientific research in a variety of fields. Exposing a semantic search layer enables natural human language to be used to discover relevant papers.
This app was built as a Single Page Application (SPA) with the following components:
Some inspiration was taken from this Cookiecutter project and turned into a SPA application instead of a separate front-end server approach.
Embeddings represent the semantic properies of the raw text and enable vector similarity search. This applications supports HuggingFace
, OpenAI
, and Cohere
embeddings out of the box.
Provider | Embedding Model | Required? |
---|---|---|
HuggingFace | sentence-transformers/all-mpnet-base-v2 |
Yes |
OpenAI | text-embedding-ada-002 |
Yes |
Cohere | embed-multilingual-v3.0 |
Yes |
Interested in a different embedding provider? Feel free to open a PR and make a suggested addition.
Want to use a different model than the one listed? Set the following environment variables in your .env
file (see below) to change:
SENTENCE_TRANSFORMER_MODEL
OPENAI_EMBEDDING_MODEL
COHERE_EMBEDDING_MODEL
$ git clone https://github.com/RedisVentures/redis-arXiv-search.git
.env.template
file:
$ cd redis-arXiv-search/
$ cp .env.template .env
OPENAI_API_KEY
to the .env
file. Need one? Get an API key
COHERE_API_KEY
to the .env
file. Need one? Get an API key
Using Redis Stack locally doesn't require any additional steps. However, it will consume more resources on your machine and have performance limitations.
Use the provided docker-compose
file for running the application locally:
$ docker compose -f docker-local-redis.yml up
Get a FREE Redis Cloud Database. Make sure to include the Search module.
Add the REDIS_HOST
, REDIS_PASSWORD
, and REDIS_PORT
environment variables to your .env
file.
Run the App:
$ docker compose -f docker-cloud-redis.yml up
data/
directory to create paper embeddings and metadata. The output JSON files will end up stored in the data/
directory and used when creating your own container../build.sh
script to build your own docker image based on the application source code and dataset changes.It's typically easier to build front end in an interactive environment, testing changes in realtime.
npm
to install yarn
)
$ cd frontend/
$ yarn install --no-optional
yarn
to serve the application from your machine
$ yarn start
http://localhost:3000
in a browser.All changes to your frontend code will be reflected in your display in semi realtime.
Every once and a while you need to clear out some Docker cached artifacts. Run docker system prune
, restart Docker Desktop, and try again.
This project is maintained by Redis on a good faith basis. Please, open an issue here on GitHub and we will try to be responsive to these.