Txtai Versions Save

This release adds dynamic embeddings vector support along with semantic graph and RAG improvements

See below for full details on the new features, improvements and bug fixes.

New Features

• Add support for dynamic vector dimensions (#674)
• Add batch node and edge creation for graphs (#693)
• Add notebook on Retrieval Augmented and Guided Generation (#694)

Improvements

• Pass options to underlying vector models (#675)
• Move vector model caching from Embeddings to Vectors (#678)
• Add indexids only search (#691)
• Create temporary tables once per database session (#692)

Bug Fixes

• Fix token storage auth error (#676)
• TypeError: 'NoneType' object is not iterable (#683)
• Fix issue with hardcoded autoawq version in example notebooks (#686)
• API deps missing Pillow (#690)

🎉 We're excited to announce the release of txtai 7.0 🎉

If you like txtai, please remember to give it a ⭐!

7.0 introduces the next generation of the semantic graph. This release adds support for graph search, advanced graph traversal and graph RAG. It also adds binary support to the API, index format improvements and training LoRA/QLoRA models. See below for more.

New Features

• Add indexing of embeddings graph relationships (#525)
• Expand the graph capabilities to enable advanced graph traversal (#534, #540)
• Add feature to return embeddings search results as graph (#644)
• Add RAG with Semantic Graphs notebook (#645)
• Graph search results via API (#670)
• Add knowledge graphs via LLM-driven entity extraction notebook (#671)
• Add advanced RAG with graph path traversal notebook (#672)
• Add support for binary content via API (#630)
• Add MessagePack encoding to API (#658)
• Add documentation for API security (#627)
• Add notebook that covers API authorization and authentication (#628)
• Add top level import for LLM (#648)
• Add external vectorization notebook (#651)
• Add configuration override to embeddings.load (#657)
• Add what's new in txtai 7.0 notebook (#673)

Improvements

• Benchmark script improvements (#641)
• ImportError: Textractor pipeline is not available - install "pipeline" extra to enable (#646)
• Resolve external vector transform functions (#650)
• Change default embeddings config format to json (#652)
• Store index ids outside of configuration when content is disabled (#653)
• Update HFTrainer to add PEFT support (#654)
• Update 40_Text_to_Speech_Generation.ipynb (#666)- thank you @babinux
• Adding training dependencies to notebooks (#669)

Bug Fixes

• Fix various issues with subindex reloading (#618)
• Fix benchmarks script (#636)
• Set tokenizer.pad_token when empty for all training paths (#649)
• Fix documentation code filters (#656)
• Issues with NetworkX when using graph subindex (#664)

A big thank you goes to Jordan Matelsky (@j6k4m8) for his help in integrating the GrandCypher library into txtai!

This release adds new LLM inference methods, API Authorization and RAG improvements

📄 New LLM methods. llama.cpp and LiteLLM support added. LLM pipeline now supports Hugging Face models, GGUF files and LLM API inference all with one line of code.

🔒 API Authorization. Adds support for API keys and pluggable authentication methods when running through txtai API.

See below for full details on the new features, improvements and bug fixes.

New Features

• Add llama.cpp support to LLM (#611)
• Integrate with Litellm (#554)
• Add API route dependencies (#623)
• Add API Authorization (#263, #624)
• Add notebook on how to build RAG pipelines (#605)
• Add notebook showing how to use llama.cpp, LiteLLM and custom generation models (#615)

Improvements

• Enhance textractor to better support RAG use cases (#603)
• Update text extraction notebook (#604)
• Extractor (RAG) pipeline improvements (#613)
• Refactor LLM pipeline to support multiple framework methods (#614)
• Change API startup event to lifespan event (#625)

Bug Fixes

• Handle None input properly in Tokenizer (#607)
• Issue with subdirectories and ZIP compression (#609)
• Error in 52_Build_RAG_pipelines_with_txtai.ipynb (#620)
• Add missing skl2onnx dependency (#622)

This release adds binary quantization, bind parameters for multimedia SQL queries and performance improvements

⚡ Scalar quantization. Supports 1 bit (binary) through 8 bit quantization. Can dramatically reduce vector storage requirements.

🚀 SQL bind parameters. Enables searching binary content with SQL statements, along with being a standard best practice.

See below for full details on the new features, improvements and bug fixes.

New Features

• Add scalar quantization support to vectors (#583)
• Feature request: Bind variable support when searching with SQL using Content=True mode (#564)
• Add cls pooling option (#565)
• Add prefix parameter for object storage (#568)
• Add parameter to RetrieveTask to disable directory flattening (#569)
• Add support for binary indexes to Faiss ANN (#585)
• Add support for scalar data to torch and numpy ANN backends (#587)
• Add quantization notebook (#588)
• Add API extensions notebook (#591)
• Add env variable to disable macOS MPS devices (#592)

Improvements

• Allow searching for images (#404)
• Update LLM pipeline to support template parameter (#566)
• Update recommended models (#573)
• Is it possible to add chat history to extractor workflow? (#575)
• Extractor pipeline improvements (#577)
• Update documentation (#582)
• Move vector normalization to vectors module (#584)
• Update benchmarks to read configuration (#586)
• Update torch version in Dockerfile (#589)
• Update Faiss ANN to support IVF strings without number of cells (#594)
• Update documentation to note SQL bind parameters (#596)

Bug Fixes

• Inconsistency in Embeddings behavior in Applications (#571)

This release adds metadata support for client-server databases and custom scoring implementations

🗃️ Client-server database integration. Store index metadata in Postgres, MariaDB/MySQL, MSSQL and more.

🖹 Custom scoring implementations. Store keyword index data in systems such as Elasticsearch. Similar to functionality already available in vector index component.

See below for full details on the new features, improvements and bug fixes.

New Features

• Add metadata support for client-server databases (#532)
• Add support for custom scoring instances (#544)
• Add benchmark script (#522)
• Add sparse keyword benchmark notebook (#523)
• Add hybrid search notebook (#526)
• Add way to load database connection URL via environment variable (#548)
• Add external database integration notebook (#549)
• Add weights and index to Application methods (#561)

Improvements

• Refresh introducing txtai notebook (#520)
• Calling .reindex() on Application instance (#547)
• Document how to run API via HTTPS (#553)
• Update reindex action to support new 6.x configuration (#557)

Bug Fixes

• ValueError: dictionary update sequence element #0 has length X; 2 is required (#529)
• Add build script workaround for DuckDB and Pandas 2.1.0 incompatibility (#542)
• Align API parameter data type to translation pipeline (#550)
• Summary pipeline error when gpu enabled on mps device (#551)
• Remove deprecated option from quantize_dynamic (#562)
• Dates fail in example (#563)

🥳 We're excited to announce the release of txtai 6.0 🥳

This significant milestone release marks txtai's 3 year birthday🎉 If you like txtai, please remember to give it a ⭐!

6.0 adds sparse, hybrid and subindexes to the embeddings interface. It also makes significant improvements to the LLM pipeline workflow. See below for more.

Breaking changes

The vast majority of changes are fully backwards compatible. New features are only enabled when specified. The only breaking change is with the Scoring terms interface, where the index format changed. The main Scoring interface used for word vectors weighting is unchanged.

New Features

• Better BM25 (#508)
• Hybrid Search (#509)
• Add additional indexes for embeddings (#515)
• Refactor Sequences and Generator pipeline into single LLM pipeline (#494)
• Support passing model parameters in pipelines (#500)
• Add "auto-id" capability to Embeddings (#502)
• Add UUID auto-id (#505)
• Add keyword arguments to Embeddings constructor (#503)
• Add top level imports (#514)

Improvements

• Add NumPy ANN Backend (#468)
• Add PyTorch ANN Backend (#469)
• Add notebook covering embeddings configuration options (#470)
• make data - No such file or directory (#473)
• Improve derivation of default embeddings model path (#476)
• Add accelerate dependency (#477)
• Add baseball example application (#484)
• Update minimum Python version to 3.8 (#485)
• Add WAL option for SQLite (#488)
• Add support for alternative acceleration devices (#489)
• Add support for passing torch devices to embeddings and pipelines (#492)
• Documentation updates (#495)
• Improve Pooling tokenizer load method (#499)
• Add ability for extractor to reference another pipeline in applications (#501)
• Reorganize embeddings configuration documentation (#504)
• Support Unicode Text Segmentation in Tokenizer (#507)
• ANN improvements (#510)
• Add multilingual graph topic modeling (#511)
• Add support for configurable text/object fields (#512)
• Update documentation for 6.0 (#513)
• Add count method to database (#517)
• Improvements when indexing through Applications (#518)
• Add what's new in txtai 6.0 notebook (#519)

Bug Fixes

• OpenMP issues with torch 1.13+ on macOS (#377)
• Unique constrant violation issue with DuckDB (#475)
• Incorrect results can be returned by embedding search when Content storage enabled (#496)
• Fix issues with graph.infertopics (#516)

This release adds the following new features, improvements and bug fixes.

Bug Fixes

• Wrap DuckDB as conditional import (#464)

This release adds workflow streams and DuckDB as a database backend

↪️️ Workflow streams enable server-side processing of large datasets. Streams iteratively pass content to workflows, no need to pass bulk data through the API.

🦆 DuckDB is a new database backend. Certain larger non-vector driven queries and aggregations will now run significantly faster than with SQLite.

See below for full details on the new features, improvements and bug fixes.

New Features

• Add workflow streams (#461)
• Add DuckDB support (#462)

Improvements

• Modify translation pipeline langdetect parameter to accept language detection function good first issue - Thank you @saucam! (#423, #444)
• Pass generation keyword arguments to underlying text generation pipeline (#457)
• Replace original prompt in text generation pipeline (#459)

Bug Fixes

• Issue with upsert and graph (#421)
• Upsert API fails with graph config while performing after /delete (#435)
• Build errors with latest onnxmltools package (#449)
• Fix issue with embeddings reindex and stale function references (#453)
• Problem with the workflow builder (#454)
• Check for empty queue before attempting to convert inputs to dictionaries (#456)
• Fix issue with latest version of Transformers and TokenDetection.save_pretrained (#458)

This release adds prompt templates, conversational task chaining and Hugging Face Hub integration

📃 Prompt templates dynamically generate text using workflow task inputs. This enables chaining multiple prompts and models together.

🤗 Embeddings now integrate with the Hugging Face Hub! Easily share and load embeddings indexes. There is a full embeddings index available for English Wikipedia.

See below for full details on the new features, improvements and bug fixes.

New Features

• Add translation pipeline parameter to return selected models and detected language - Thank you @saucam! (#383, #424)
• Add sample parameter to Faiss ANN (#427)
• Add support for instruction-based embeddings (#428)
• Add Hugging Face Hub integration (#430)
• Add cloud object storage support for uncompressed embeddings indexes (#431)
• Add support for custom cloud providers (#432)
• Add support for storing embeddings config as JSON (#433)
• Add notebook for syncing embeddings with the cloud (#434)
• Add terms method to embeddings (#445)
• Add extractor reference output format (#446)
• Add template task (#448)
• Add prompt template and task chaining example notebook (#451)

Improvements

• Mention the default storage engine - Thank you @hsm207! (#422)
• Refactor archive module into separate package (#429)
• Resolve application references in pipelines (#441)
• Extractor pipeline improvements (#443)
• Allow task action arguments to be dictionaries in addition to tuples (#447)
• Automatically mark embeddings index files for lfs tracking with Hugging Face Hub (#450)

Bug Fixes

• Pin onnxruntime for macOS in build script (#425)

This release adds embeddings-guided and prompt-driven search along with a number of methods to train language models

🔎 Prompt-driven search is a big step forward towards conversational search in txtai. With this release, complex prompts can now be passed to txtai to customize how search results are returned. Lots of exciting possibilities on this front, stay tuned.

💡 The trainer pipeline now has support for training language models from scratch. It supports masked language modeling (MLM), causal language modeling (CLM) and replaced token detection (ELECTRA-style). This is part of the micromodels effort.

See below for full details on the new features, improvements and bug fixes.

New Features

• Add language modeling task to HFTrainer (#403)
• Add language modeling example notebook (#408)
• Add FAQ section to documentation (#413)
• Add language generation task to HFTrainer (#414)
• Add replaced token detection task to HFTrainer (#415)
• Add generator pipeline for text generation (#416)
• Add notebook for embeddings-guided and prompt-driven search with LLMs (#418)

Improvements

• Normalize BM25 and TF-IDF scores (#401)
• Add note to restart kernel if running in Google Colab - Thank you @hsm207! (#410)
• Add clear error when starting API and config file not found (#412)
• Extractor pipeline 2.0 (#417)
• Make texts parameter optional for extractor pipeline in applications (#420)

Bug Fixes

• Fix issue with ORDER BY case sensitivity (#405)