Rapidsai Raft Versions Save

RAFT contains fundamental widely-used algorithms and primitives for machine learning and information retrieval. The algorithms are CUDA-accelerated and form building blocks for more easily writing high performance applications.

v24.04.00

1 month ago

🐛 Bug Fixes

Update pre-commit-hooks to v0.0.3 (#2239) @KyleFromNVIDIA
MAINT: Simplify NCCL worker rank identification (#2228) @VibhuJawa
Fix bug in blockRankedReduce (#2226) @akifcorduk
Fix illegal acces mean/stdev, sum add Kahan Summation (#2223) @mfoerste4
Batch cutlass distance kernels along N matrix dim (#2215) @mdoijade
Fix out of bounds access in sum kernel (#2183) @tfeher
Fix ANN bench ground truth generation for k>1024 (#2180) @tfeher
Fixing cusparse aligned address issue and adding note (#2179) @cjnolet
Launch neighborhood_recall kernel on CUDA stream (#2156) @divyegala
Add compile-library by default on pylibraft build (#2090) @lowener

📖 Documentation

Adding cuVS notice to README and front page of docs. (#2224) @cjnolet

🚀 New Features

Add CAGRA-Q to ANN benchmarks (#2233) @achirkin
Add CAGRA-Q build (compression) (#2213) @achirkin
CAGRA-Q search (#2206) @enp1s0
Demangle backtrace symbols on raft error (#2188) @achirkin
Reapply: Support for fp16 in CAGRA and IVF-PQ (#2172) @achirkin
Remove supports_streams from custom RAFT memory resources (#2121) @harrism
[FEA] Add support for bitmap_view & the API of bitmap_to_csr (#2109) @rhdong

🛠️ Improvements

Use conda env create --yes instead of --force (#2247) @bdice
Align ucx version pinning with ucx-py/ucxx. (#2227) @bdice
Add upper bound to prevent usage of NumPy 2 (#2222) @bdice
Performance optimization of IVF-flat / select_k (#2221) @mfoerste4
Replace local copyright check with pre-commit-hooks verify-copyright (#2220) @KyleFromNVIDIA
Remove hard-coding of RAPIDS version where possible (#2219) @KyleFromNVIDIA
Fix style. (#2214) @bdice
Add explicit instantiations for IVF-PQ search kernels used in tests (#2212) @tfeher
Improve RBC eps-neighborhood query performance (#2211) @mfoerste4
Add test for spmm (#2210) @mfoerste4
Only install necessary components in conda packages. (#2209) @bdice
Automate C++ include file grouping and ordering using clang-format (#2202) @harrism
Add support for Python 3.11, require NumPy 1.23+ (#2200) @jameslamb
Pass std::optional instead of thrust::optional to RMM (#2199) @trxcllnt
Update devcontainers to CUDA Toolkit 12.2 (#2192) @trxcllnt
target branch-24.04 for GitHub Actions workflows (#2189) @jameslamb
Fixing workaround for cuSPARSE bug with correct copy dimensions (#2185) @mfoerste4
Allow topk larger than 1024 in CAGRA (#2181) @benfred
IVF-FLAT support k > 256 (#2169) @mfoerste4
Add environment-agnostic scripts for running ctests and pytests (#2165) @trxcllnt
Ensure that ctest is called with --no-tests=error. (#2163) @bdice
Update ops-bot.yaml (#2158) @AyodeAwe
random sampling of dataset rows with improved memory utilization (#2155) @tfeher
[FIX] Ensure hnswlib can be found from RAFT's build dir (#2145) @trxcllnt
Improve analysis experience for ANN benchmarks (#2139) @achirkin
Enable CAGRA index building without adding dataset to the index (#2126) @tfeher
Add fused cosine 1-NN cutlass based kernel (#2125) @mdoijade
Update raft for compatibility with the latest cuco (#2118) @PointKernel
Support CUDA 12.2 (#2092) @jameslamb
Cache IVF-PQ and select-warpsort kernel launch parameters to reduce latency (#1786) @achirkin

v24.06.00a

1 month ago

🔗 Links

🚨 Breaking Changes

Convert device_memory_resource* to device_async_resource_ref (#2269) @harrism

🐛 Bug Fixes

Fix build command for C++ compilation (#2270) @lowener
Fix a compilation error in CAGRA when enabling log output (#2262) @enp1s0
Correct member initialization order (#2254) @robertmaynard
Fix time computation in CAGRA notebook (#2231) @lowener

🚀 New Features

Always use a static gtest and gbench (#2265) @robertmaynard
InnerProduct Distance Metric for CAGRA search (#2260) @tarang-jain
[FEA] Add support for select_k on CSR matrix (#2140) @rhdong

🛠️ Improvements

Use dynamic version for raft-ann-bench (#2285) @KyleFromNVIDIA
Make 'librmm' a 'host' dependency for conda packages (#2284) @jameslamb
Fix comments in cpp/include/raft/neighbors/cagra_serialize.cuh (#2283) @jiangyinzuo
define 'ucx' pytest marker (#2281) @jameslamb
add --rm and --name to devcontainer run args (#2275) @trxcllnt
Update pip devcontainers to UCX v1.15.0 (#2274) @trxcllnt
#ifdef out pragma deprecation warning messages (#2271) @trxcllnt
Convert device_memory_resource* to device_async_resource_ref (#2269) @harrism
Update the developer's guide with new copyright hook (#2266) @KyleFromNVIDIA
Improve coalesced reduction performance for tall and thin matrices (up to 2.6x faster) (#2259) @Nyrio
Enable all tests for arm64 jobs (#2248) @galipremsagar
Add CAGRA-Q subspace dim = 4 support (#2244) @enp1s0
Get rid of cuco::sentinel namespace (#2243) @PointKernel
Replace usages of raw get_upstream with get_upstream_resource() (#2207) @miscco

v24.02.00

3 months ago

🚨 Breaking Changes

Switch to scikit-build-core (#2051) @vyasr
Update to CCCL 2.2.0. (#2049) @bdice
Update raft-ann-bench output filenames and add features to plotting (#2043) @divyegala
Remove selection_faiss (#2027) @benfred

🐛 Bug Fixes

fix is_row/col_order for strided layouts (#2173) @mfoerste4
Fix failing C++ tests and revert #2097, #2085. (#2168) @cjnolet
Exclude tests from builds (#2162) @vyasr
[HOTFIX] 24.02 Revert Random Sampling (#2144) @cjnolet
Pin to pytest 7. (#2137) @bdice
Conditionally include hnsw wrapper source in CMake (#2135) @divyegala
[BUG] Fix SPMM strided view (#2124) @lowener
Fixing small bug in CUSPARSE spmm w/ CUDA 12.2 (#2117) @cjnolet
[BUG] Fix num_cta_per_query div (#2107) @lowener
Remove extraneous host pinnings from libraft-headers-only. (#2102) @bdice
Remove unneeded CI symbol excludes (#2098) @robertmaynard
Properly taking ownership of nccl subcomm (and destroying it) (#2094) @cjnolet
Fix max_queries for CAGRA (#2081) @lowener
Fix compile failure on RTX 4090 (#2076) @JieFengWang
Fix a crash in FAISS benchmark wrapper introduced in #2021 (#2062) @achirkin
Correct function that wasn't returning a value (#2045) @robertmaynard
Fixing small bug in raft-ann-bench (#2041) @cjnolet
Make device_resources accessed from device_resources_manager thread-safe (#2030) @wphicks
Fix ann-bench multithreading (#2021) @achirkin
Fix ci/checks/copyright.py to mirror RAPIDS reference (#2008) @divyegala
Fix pyproject versions (#2002) @vyasr

📖 Documentation

Adding license info for wiki-all dataset (#2129) @cjnolet
[DOC] Documentation updates for release 24.02 (#2093) @cjnolet
Fix errors with ingroup exposed by doxygen 1.10 (#2079) @wphicks
Fix a typo (#2070) @narangvivek10
Add usage example for brute_force::build (#2029) @benfred
Add filtering to vector search tutorial (#1996) @lowener

🚀 New Features

Update to use rapids-cmake for all deps (#2096) @robertmaynard
Add IVF-PQ example into the template project (#2091) @achirkin
Support for fp16 in CAGRA and IVF-PQ (#2085) @achirkin
Add random subsampling for IVF methods (#2077) @tfeher
Update raft-ann-bench output filenames and add features to plotting (#2043) @divyegala
Add brute_force index serialization (#2036) @wphicks
Add eps-neighbor search via RBC (#2028) @mfoerste4
libraft and pylibraft API for CAGRA build and HNSW search (#2022) @divyegala
Export Pareto frontier in raft-ann-bench.data_export (#2009) @divyegala
Implement maybe-owning multi-dimensional container (mdbuffer) (#1999) @wphicks
Add support for 1024+ dim vectors in CAGRA search (#1994) @enp1s0
Replace GEMM backend: cublas.gemm -> cublaslt.matmul (#1736) @achirkin

🛠️ Improvements

Remove get_mem_info functions from RAFT custom memory resources (#2108) @harrism
Replace call to mr::get_mem_info() (#2099) @harrism
Allow topk larger than 1024 in CAGRA (#2097) @benfred
Remove usages of rapids-env-update (#2095) @KyleFromNVIDIA
Provide explicit pool size for pool_memory_resources and clean up includes (#2088) @harrism
refactor CUDA versions in dependencies.yaml (#2086) @jameslamb
ANN bench fix latency measurement overhead (#2084) @tfeher
Remove hardcoded limit in print_results function (#2080) @narangvivek10
[FEA] Add support for SDDMM by wrapping the cusparseSDDMM (#2067) (#2067) @rhdong
Benchmark brute force knn (#2063) @benfred
[BUG] fix empty initialization of device_ndarray in pylibraft (#2061) @mfoerste4
Improve parallelism of refine host (#2059) @anaruse
Subsampling for IVF-PQ codebook generation (#2052) @abc99lr
Switch to scikit-build-core (#2051) @vyasr
Update to CCCL 2.2.0. (#2049) @bdice
Use cuda::proclaim_return_type on device lambda. (#2048) @bdice
Removing code that explicitly compares equality of rmm memory resources (#2047) @cjnolet
Add public enum for select-k algorithm selection (#2046) @benfred
Update dependencies.yaml to new pip index (#2042) @vyasr
Remove RAFT_BUILD_WHEELS and standardize Python builds (#2040) @vyasr
Fix ucx-py version pinning in dependencies.yaml. (#2035) @bdice
[REVIEW] Fix typos in parameter tuning guide (#2034) @abc99lr
Add AIR-Top-k reference (#2031) @tfeher
Remove selection_faiss (#2027) @benfred
Fixing json parse error in raft-ann-bench.data_export (#2025) @cjnolet
Updating cagra build constraint (#2016) @cjnolet
Update to fmt 10.1.1 and spdlog 1.12.0. (#1957) @bdice
Enable host dataset for IVF-Flat (#1635) @tfeher
add half/bfloat support to myInf and abs (#1592) @Kh4ster

v24.04.00a

3 months ago

🔗 Links

🐛 Bug Fixes

Update pre-commit-hooks to v0.0.3 (#2239) @KyleFromNVIDIA
MAINT: Simplify NCCL worker rank identification (#2228) @VibhuJawa
Fix bug in blockRankedReduce (#2226) @akifcorduk
Fix illegal acces mean/stdev, sum add Kahan Summation (#2223) @mfoerste4
Batch cutlass distance kernels along N matrix dim (#2215) @mdoijade
Fix out of bounds access in sum kernel (#2183) @tfeher
Fix ANN bench ground truth generation for k>1024 (#2180) @tfeher
Fixing cusparse aligned address issue and adding note (#2179) @cjnolet
Launch neighborhood_recall kernel on CUDA stream (#2156) @divyegala
Add compile-library by default on pylibraft build (#2090) @lowener

📖 Documentation

Adding cuVS notice to README and front page of docs. (#2224) @cjnolet

🚀 New Features

Add CAGRA-Q to ANN benchmarks (#2233) @achirkin
Add CAGRA-Q build (compression) (#2213) @achirkin
CAGRA-Q search (#2206) @enp1s0
Demangle backtrace symbols on raft error (#2188) @achirkin
Reapply: Support for fp16 in CAGRA and IVF-PQ (#2172) @achirkin
Remove supports_streams from custom RAFT memory resources (#2121) @harrism
[FEA] Add support for bitmap_view & the API of bitmap_to_csr (#2109) @rhdong

🛠️ Improvements

Align ucx version pinning with ucx-py/ucxx. (#2227) @bdice
Add upper bound to prevent usage of NumPy 2 (#2222) @bdice
Performance optimization of IVF-flat / select_k (#2221) @mfoerste4
Replace local copyright check with pre-commit-hooks verify-copyright (#2220) @KyleFromNVIDIA
Remove hard-coding of RAPIDS version where possible (#2219) @KyleFromNVIDIA
Fix style. (#2214) @bdice
Add explicit instantiations for IVF-PQ search kernels used in tests (#2212) @tfeher
Improve RBC eps-neighborhood query performance (#2211) @mfoerste4
Add test for spmm (#2210) @mfoerste4
Only install necessary components in conda packages. (#2209) @bdice
Automate C++ include file grouping and ordering using clang-format (#2202) @harrism
Add support for Python 3.11, require NumPy 1.23+ (#2200) @jameslamb
Pass std::optional instead of thrust::optional to RMM (#2199) @trxcllnt
Update devcontainers to CUDA Toolkit 12.2 (#2192) @trxcllnt
target branch-24.04 for GitHub Actions workflows (#2189) @jameslamb
Fixing workaround for cuSPARSE bug with correct copy dimensions (#2185) @mfoerste4
Allow topk larger than 1024 in CAGRA (#2181) @benfred
IVF-FLAT support k > 256 (#2169) @mfoerste4
Add environment-agnostic scripts for running ctests and pytests (#2165) @trxcllnt
Ensure that ctest is called with --no-tests=error. (#2163) @bdice
Update ops-bot.yaml (#2158) @AyodeAwe
random sampling of dataset rows with improved memory utilization (#2155) @tfeher
[FIX] Ensure hnswlib can be found from RAFT's build dir (#2145) @trxcllnt
Improve analysis experience for ANN benchmarks (#2139) @achirkin
Enable CAGRA index building without adding dataset to the index (#2126) @tfeher
Add fused cosine 1-NN cutlass based kernel (#2125) @mdoijade
Update raft for compatibility with the latest cuco (#2118) @PointKernel
Support CUDA 12.2 (#2092) @jameslamb
Cache IVF-PQ and select-warpsort kernel launch parameters to reduce latency (#1786) @achirkin

v23.12.00

5 months ago

🐛 Bug Fixes

Update actions/labeler to v4 (#2037) @raydouglass
pylibraft only depends on numpy at runtime, not build time. (#2013) @bdice
Fixes to update-version.sh (#1991) @raydouglass
Adjusting end-to-end start time so it doesn't include stream creation time (#1989) @cjnolet
CAGRA graph optimizer: clamp rev_graph_count (#1987) @tfeher
Catching conversion errors in data_export instead of fully failing (#1979) @cjnolet
Fix syncing mechanism in raft-ann-bench C++ search (#1961) @divyegala
Fixing hnswlib in latency mode (#1959) @cjnolet
Fix ucx-py alpha version update for raft-dask (#1953) @divyegala
Reduce NN Descent test threshold (#1946) @divyegala
Fixes to new YAML config raft-bench-ann (#1945) @divyegala
Set RNG seeds in NN Descent to diagnose flaky tests (#1931) @divyegala
Fix FAISS CPU algorithm names in raft-ann-bench (#1916) @divyegala
Increase iterations in NN Descent tests to avoid flakiness (#1915) @divyegala
Fix filepath in raft-ann-bench/split_groundtruth module (#1911) @divyegala
Remove dynamic entry-points from raft-ann-bench (#1910) @benfred
Remove unnecessary dataset path check in ANN bench (#1908) @tfeher
Fixing Googletests and re-enabling in CI (#1904) @cjnolet
Fix NN Descent overflows (#1875) @divyegala
Build fix for CUDA 12.2 (#1870) @benfred
[BUG] Fix a bug in NN descent (#1869) @enp1s0

📖 Documentation

Brute Force Index documentation fix (#1944) @lowener
Add wiki_all dataset config and documentation. (#1918) @cjnolet
Updates to raft-ann-bench docs (#1905) @cjnolet
End-to-end vector search tutorial in docs (#1776) @cjnolet

🚀 New Features

Adding dry-run option to raft-ann-bench (#1970) @cjnolet
Add ANN bench scripts to generate ground truth (#1967) @tfeher
CAGRA build + HNSW search (#1956) @divyegala
Verify conda-cpp-post-build-checks (#1935) @robertmaynard
Make all cuda kernels have hidden visibility (#1898) @robertmaynard
Update rapids-cmake functions to non-deprecated signatures (#1884) @robertmaynard
[FEA] Helpers for identifying contiguous layouts. (#1861) @trivialfis
Add raft::stats::neighborhood_recall (#1860) @divyegala
[FEA] Helpers and CodePacker for IVF-PQ (#1826) @tarang-jain

🛠️ Improvements

Pinning fmt and spdlog for raft-ann-bench-cpu (#2018) @cjnolet
Build concurrency for nightly and merge triggers (#2011) @bdice
Using EXPORT_SET in rapids_find_package_root (#2006) @cjnolet
Remove static checks for serialization size (#1997) @cjnolet
Skipping bad json parse (#1990) @cjnolet
Update select-k heuristic (#1985) @benfred
ANN bench: use different offset for each thread (#1981) @tfeher
Allow raft-ann-bench/run to continue after encountering bad YAML configs (#1980) @divyegala
Add build and search params to raft-ann-bench.data_export CSVs (#1971) @divyegala
Use new rapids-dask-dependency metapackage for managing dask versions (#1968) @galipremsagar
Remove unused header (#1960) @wphicks
Adding pool back in and fixing cagra benchmark params (#1951) @cjnolet
Add constraints to hnswlib in raft-bench-ann (#1949) @divyegala
Add support for iterating over batches in bfknn (#1947) @benfred
Fix ANN bench latency (#1940) @tfeher
Add YAML config files to run parameter sweeps for ANN benchmarks (#1929) @divyegala
Relax ucx pinning (#1927) @vyasr
Try using contiguous rank to fix cuda_visible_devices (#1926) @VibhuJawa
Unpin dask and distributed for 23.12 development (#1925) @galipremsagar
Adding throughput and latency modes to raft-ann-bench (#1920) @cjnolet
Providing aarch64 yaml environment files (#1914) @cjnolet
CAGRA ANN bench: parse build options for IVF-PQ build algo (#1912) @tfeher
Fix python script location in ANN bench description (#1906) @tfeher
Refactor install/build guide. (#1899) @cjnolet
Check return values of raft-ann-bench subprocess calls (#1897) @benfred
ANN bench options to specify CAGRA graph and dataset locations (#1896) @cjnolet
Add check-json to pre-commit linters, and fix invalid ann-bench JSON config (#1894) @benfred
Use branch-23.12 workflows. (#1886) @bdice
Setup Consistent Nightly Versions for Pip and Conda (#1880) @divyegala
Fix and improve one-block radix select (#1878) @yong-wang
[FEA] Improvements on bitset class (#1877) @lowener
Branch 23.12 merge 23.10 (#1873) @AyodeAwe
Branch 23.12 merge 23.10 (#1868) @cjnolet
Replace raft::random calls to not use deprecated API (#1867) @lowener
raft: Build CUDA 12.0 ARM conda packages. (#1853) @bdice
Documentation for raft ANN benchmark containers. (#1833) @dantegd
[FEA] Support vector deletion in ANN IVF (#1831) @lowener
Provide a raft::copy overload for mdspan-to-mdspan copies (#1818) @wphicks
Adding FAISS cpu to raft-ann-bench (#1814) @cjnolet

v24.02.00a

5 months ago

🔗 Links

🚨 Breaking Changes

Switch to scikit-build-core (#2051) @vyasr
Update to CCCL 2.2.0. (#2049) @bdice
Update raft-ann-bench output filenames and add features to plotting (#2043) @divyegala
Remove selection_faiss (#2027) @benfred

🐛 Bug Fixes

[BUG] Fix SPMM strided view (#2124) @lowener
Fixing small bug in CUSPARSE spmm w/ CUDA 12.2 (#2117) @cjnolet
[BUG] Fix num_cta_per_query div (#2107) @lowener
Remove extraneous host pinnings from libraft-headers-only. (#2102) @bdice
Remove unneeded CI symbol excludes (#2098) @robertmaynard
Properly taking ownership of nccl subcomm (and destroying it) (#2094) @cjnolet
Fix max_queries for CAGRA (#2081) @lowener
Fix compile failure on RTX 4090 (#2076) @JieFengWang
Fix a crash in FAISS benchmark wrapper introduced in #2021 (#2062) @achirkin
Correct function that wasn't returning a value (#2045) @robertmaynard
Fixing small bug in raft-ann-bench (#2041) @cjnolet
Make device_resources accessed from device_resources_manager thread-safe (#2030) @wphicks
Fix ann-bench multithreading (#2021) @achirkin
Fix ci/checks/copyright.py to mirror RAPIDS reference (#2008) @divyegala
Fix pyproject versions (#2002) @vyasr

📖 Documentation

Adding license info for wiki-all dataset (#2129) @cjnolet
[DOC] Documentation updates for release 24.02 (#2093) @cjnolet
Fix errors with ingroup exposed by doxygen 1.10 (#2079) @wphicks
Fix a typo (#2070) @narangvivek10
Add usage example for brute_force::build (#2029) @benfred
Add filtering to vector search tutorial (#1996) @lowener

🚀 New Features

Update to use rapids-cmake for all deps (#2096) @robertmaynard
Add IVF-PQ example into the template project (#2091) @achirkin
Support for fp16 in CAGRA and IVF-PQ (#2085) @achirkin
Add random subsampling for IVF methods (#2077) @tfeher
Update raft-ann-bench output filenames and add features to plotting (#2043) @divyegala
Add brute_force index serialization (#2036) @wphicks
Add eps-neighbor search via RBC (#2028) @mfoerste4
libraft and pylibraft API for CAGRA build and HNSW search (#2022) @divyegala
Export Pareto frontier in raft-ann-bench.data_export (#2009) @divyegala
Implement maybe-owning multi-dimensional container (mdbuffer) (#1999) @wphicks
Add support for 1024+ dim vectors in CAGRA search (#1994) @enp1s0
Replace GEMM backend: cublas.gemm -> cublaslt.matmul (#1736) @achirkin

🛠️ Improvements

Remove get_mem_info functions from RAFT custom memory resources (#2108) @harrism
Replace call to mr::get_mem_info() (#2099) @harrism
Allow topk larger than 1024 in CAGRA (#2097) @benfred
Remove usages of rapids-env-update (#2095) @KyleFromNVIDIA
Provide explicit pool size for pool_memory_resources and clean up includes (#2088) @harrism
refactor CUDA versions in dependencies.yaml (#2086) @jameslamb
ANN bench fix latency measurement overhead (#2084) @tfeher
Remove hardcoded limit in print_results function (#2080) @narangvivek10
[FEA] Add support for SDDMM by wrapping the cusparseSDDMM (#2067) (#2067) @rhdong
Benchmark brute force knn (#2063) @benfred
[BUG] fix empty initialization of device_ndarray in pylibraft (#2061) @mfoerste4
Improve parallelism of refine host (#2059) @anaruse
Subsampling for IVF-PQ codebook generation (#2052) @abc99lr
Switch to scikit-build-core (#2051) @vyasr
Update to CCCL 2.2.0. (#2049) @bdice
Use cuda::proclaim_return_type on device lambda. (#2048) @bdice
Removing code that explicitly compares equality of rmm memory resources (#2047) @cjnolet
Add public enum for select-k algorithm selection (#2046) @benfred
Update dependencies.yaml to new pip index (#2042) @vyasr
Remove RAFT_BUILD_WHEELS and standardize Python builds (#2040) @vyasr
Fix ucx-py version pinning in dependencies.yaml. (#2035) @bdice
[REVIEW] Fix typos in parameter tuning guide (#2034) @abc99lr
Add AIR-Top-k reference (#2031) @tfeher
Remove selection_faiss (#2027) @benfred
Fixing json parse error in raft-ann-bench.data_export (#2025) @cjnolet
Updating cagra build constraint (#2016) @cjnolet
Update to fmt 10.1.1 and spdlog 1.12.0. (#1957) @bdice
Enable host dataset for IVF-Flat (#1635) @tfeher
add half/bfloat support to myInf and abs (#1592) @Kh4ster

v23.10.00

7 months ago

🚨 Breaking Changes

Change CAGRA auto mode selection (#1830) @enp1s0
Update CAGRA serialization (#1755) @benfred
Improvements to ANN Benchmark Python scripts and docs (#1734) @divyegala
Update to Cython 3.0.0 (#1688) @vyasr
ANN-benchmarks: switch to use gbench (#1661) @achirkin

🐛 Bug Fixes

[BUG] Fix a bug in the filtering operation in CAGRA multi-kernel (#1862) @enp1s0
Fix conf file for benchmarking glove datasets (#1846) @dantegd
raft-ann-bench package fixes for plotting and conf files (#1844) @dantegd
Fix update-version.sh for all pyproject.toml files (#1839) @raydouglass
Make RMM a run dependency of the raft-ann-bench conda package (#1838) @dantegd
Printing actual exception in require base set (#1816) @cjnolet
Adding rmm to raft-ann-bench dependencies (#1815) @cjnolet
Use conda mambabuild not mamba mambabuild (#1812) @bdice
Fix raft-dask naming in wheel builds (#1805) @divyegala
neighbors::refine_host: check the dataset bounds (#1793) @achirkin
[BUG] Fix search parameter check in CAGRA (#1784) @enp1s0
IVF-Flat: fix search batching (#1764) @achirkin
Using expanded distance computations in pylibraft (#1759) @cjnolet
Fix ann-bench Documentation (#1754) @divyegala
Make get_cache_idx a weak symbol with dummy template (#1733) @ahendriksen
Fix IVF-PQ fused kernel performance problems (#1726) @achirkin
Fix build.sh to enable NEIGHBORS_ANN_CAGRA_TEST (#1724) @enp1s0
Fix template types for create_descriptor function. (#1680) @csadorf

📖 Documentation

Fix the CAGRA paper citation (#1788) @enp1s0
Add citation info for the CAGRA paper preprint (#1787) @enp1s0
[DOC] Fix grouping for ANN in C++ doxygen (#1782) @lowener
Update RAFT documentation (#1717) @lowener
Additional polishing of README and docs (#1713) @cjnolet

🚀 New Features

[FEA] Add bitset_filter for CAGRA indices removal (#1837) @lowener
ann-bench: miscellaneous improvements (#1808) @achirkin
[FEA] Add bitset for ANN pre-filtering and deletion (#1803) @lowener
Adding config files for remaining (relevant) ann-benchmarks million-scale datasets (#1761) @cjnolet
Port NN-descent algorithm to use in cagra::build() (#1748) @divyegala
Adding conda build for libraft static (#1746) @cjnolet
[FEA] Provide device_resources_manager for easy generation of device_resources (#1716) @wphicks

🛠️ Improvements

Add option to brute_force index to store non-owning reference to norms (#1865) @benfred
Pin dask and distributed for 23.10 release (#1864) @galipremsagar
Update image names (#1835) @AyodeAwe
Fixes for OOM during CAGRA benchmarks (#1832) @benfred
Change CAGRA auto mode selection (#1830) @enp1s0
Update to clang 16.0.6. (#1829) @bdice
Add IVF-Flat C++ example (#1828) @tfeher
matrix::select_k: extra tests and benchmarks (#1821) @achirkin
Add index class for brute_force knn (#1817) @benfred
[FEA] Add pre-filtering to CAGRA (#1811) @enp1s0
More updates to ann-bench docs (#1810) @cjnolet
Add best deep-100M configs for IVF-PQ to ANN benchmarks (#1807) @tfeher
A few fixes to raft-ann-bench recipe and docs (#1806) @cjnolet
Simplify wheel build scripts and allow alphas of RAPIDS dependencies (#1804) @divyegala
Various fixes to reproducible benchmarks (#1800) @cjnolet
ANN-bench: more flexible cuda_stub.hpp (#1792) @achirkin
Add RAFT devcontainers (#1791) @trxcllnt
Cagra memory optimizations (#1790) @benfred
Fixing a couple security concerns in raft-dask nccl unique id generation (#1785) @cjnolet
Don't serialize dataset with CAGRA bench (#1781) @benfred
Use copy-pr-bot (#1774) @ajschmidt8
Add GPU and CPU packages for ANN benchmarks (#1773) @dantegd
Improvements to raft-ann-bench scripts, docs, and benchmarking implementations. (#1769) @cjnolet
[REVIEW] Introducing host API for PCG (#1767) @vinaydes
Unpin dask and distributed for 23.10 development (#1760) @galipremsagar
Add ivf-flat notebook (#1758) @tfeher
Update CAGRA serialization (#1755) @benfred
Remove block size template parameter from CAGRA search (#1740) @enp1s0
Add NVTX ranges for cagra search/serialize functions (#1737) @benfred
Improvements to ANN Benchmark Python scripts and docs (#1734) @divyegala
Fixing forward merger for 23.08 -> 23.10 (#1731) @cjnolet
[FEA] Use CAGRA in C++ template (#1730) @lowener
fixed box around raft image (#1710) @nwstephens
Enable CUTLASS-based distance kernels on CTK 12 (#1702) @ahendriksen
Update bench-ann configuration (#1696) @lowener
Update to Cython 3.0.0 (#1688) @vyasr
Update CMake version (#1677) @vyasr
Branch 23.10 merge 23.08 (#1672) @vyasr
ANN-benchmarks: switch to use gbench (#1661) @achirkin

v23.12.00a

7 months ago

🔗 Links

🐛 Bug Fixes

pylibraft only depends on numpy at runtime, not build time. (#2013) @bdice
Fixes to update-version.sh (#1991) @raydouglass
Adjusting end-to-end start time so it doesn't include stream creation time (#1989) @cjnolet
CAGRA graph optimizer: clamp rev_graph_count (#1987) @tfeher
Catching conversion errors in data_export instead of fully failing (#1979) @cjnolet
Fix syncing mechanism in raft-ann-bench C++ search (#1961) @divyegala
Fixing hnswlib in latency mode (#1959) @cjnolet
Fix ucx-py alpha version update for raft-dask (#1953) @divyegala
Reduce NN Descent test threshold (#1946) @divyegala
Fixes to new YAML config raft-bench-ann (#1945) @divyegala
Set RNG seeds in NN Descent to diagnose flaky tests (#1931) @divyegala
Fix FAISS CPU algorithm names in raft-ann-bench (#1916) @divyegala
Increase iterations in NN Descent tests to avoid flakiness (#1915) @divyegala
Fix filepath in raft-ann-bench/split_groundtruth module (#1911) @divyegala
Remove dynamic entry-points from raft-ann-bench (#1910) @benfred
Remove unnecessary dataset path check in ANN bench (#1908) @tfeher
Fixing Googletests and re-enabling in CI (#1904) @cjnolet
Fix NN Descent overflows (#1875) @divyegala
Build fix for CUDA 12.2 (#1870) @benfred
[BUG] Fix a bug in NN descent (#1869) @enp1s0

📖 Documentation

Brute Force Index documentation fix (#1944) @lowener
Add wiki_all dataset config and documentation. (#1918) @cjnolet
Updates to raft-ann-bench docs (#1905) @cjnolet
End-to-end vector search tutorial in docs (#1776) @cjnolet

🚀 New Features

Adding dry-run option to raft-ann-bench (#1970) @cjnolet
Add ANN bench scripts to generate ground truth (#1967) @tfeher
CAGRA build + HNSW search (#1956) @divyegala
Verify conda-cpp-post-build-checks (#1935) @robertmaynard
Make all cuda kernels have hidden visibility (#1898) @robertmaynard
Update rapids-cmake functions to non-deprecated signatures (#1884) @robertmaynard
[FEA] Helpers for identifying contiguous layouts. (#1861) @trivialfis
Add raft::stats::neighborhood_recall (#1860) @divyegala
[FEA] Helpers and CodePacker for IVF-PQ (#1826) @tarang-jain

🛠️ Improvements

Pinning fmt and spdlog for raft-ann-bench-cpu (#2018) @cjnolet
Build concurrency for nightly and merge triggers (#2011) @bdice
Using EXPORT_SET in rapids_find_package_root (#2006) @cjnolet
Remove static checks for serialization size (#1997) @cjnolet
Skipping bad json parse (#1990) @cjnolet
Update select-k heuristic (#1985) @benfred
ANN bench: use different offset for each thread (#1981) @tfeher
Allow raft-ann-bench/run to continue after encountering bad YAML configs (#1980) @divyegala
Add build and search params to raft-ann-bench.data_export CSVs (#1971) @divyegala
Use new rapids-dask-dependency metapackage for managing dask versions (#1968) @galipremsagar
Remove unused header (#1960) @wphicks
Adding pool back in and fixing cagra benchmark params (#1951) @cjnolet
Add constraints to hnswlib in raft-bench-ann (#1949) @divyegala
Add support for iterating over batches in bfknn (#1947) @benfred
Fix ANN bench latency (#1940) @tfeher
Add YAML config files to run parameter sweeps for ANN benchmarks (#1929) @divyegala
Relax ucx pinning (#1927) @vyasr
Try using contiguous rank to fix cuda_visible_devices (#1926) @VibhuJawa
Unpin dask and distributed for 23.12 development (#1925) @galipremsagar
Adding throughput and latency modes to raft-ann-bench (#1920) @cjnolet
Providing aarch64 yaml environment files (#1914) @cjnolet
CAGRA ANN bench: parse build options for IVF-PQ build algo (#1912) @tfeher
Fix python script location in ANN bench description (#1906) @tfeher
Refactor install/build guide. (#1899) @cjnolet
Check return values of raft-ann-bench subprocess calls (#1897) @benfred
ANN bench options to specify CAGRA graph and dataset locations (#1896) @cjnolet
Add check-json to pre-commit linters, and fix invalid ann-bench JSON config (#1894) @benfred
Use branch-23.12 workflows. (#1886) @bdice
Setup Consistent Nightly Versions for Pip and Conda (#1880) @divyegala
Fix and improve one-block radix select (#1878) @yong-wang
[FEA] Improvements on bitset class (#1877) @lowener
Branch 23.12 merge 23.10 (#1873) @AyodeAwe
Branch 23.12 merge 23.10 (#1868) @cjnolet
Replace raft::random calls to not use deprecated API (#1867) @lowener
raft: Build CUDA 12.0 ARM conda packages. (#1853) @bdice
Documentation for raft ANN benchmark containers. (#1833) @dantegd
[FEA] Support vector deletion in ANN IVF (#1831) @lowener
Provide a raft::copy overload for mdspan-to-mdspan copies (#1818) @wphicks
Adding FAISS cpu to raft-ann-bench (#1814) @cjnolet

v23.08.00

9 months ago

🚨 Breaking Changes

Separate CAGRA index type from internal idx type (#1664) @tfeher
Stop using setup.py in build.sh (#1645) @vyasr
CAGRA max_queries auto configuration (#1613) @enp1s0
Rename the CAGRA prune function to optimize (#1588) @enp1s0
CAGRA pad dataset for 128bit vectorized load (#1505) @tfeher
Sparse Pairwise Distances API Updates (#1502) @divyegala
Cagra index construction without copying device mdarrays (#1494) @tfeher
[FEA] Masked NN for connect_components (#1445) @tarang-jain
Limiting workspace memory resource (#1356) @achirkin

🐛 Bug Fixes

Remove push condition on docs-build (#1693) @raydouglass
IVF-PQ: Fix illegal memory access with large max_samples (#1685) @achirkin
Fix missing parameter for select_k (#1682) @ucassjy
Separate CAGRA index type from internal idx type (#1664) @tfeher
Add rmm to pylibraft run dependencies, since it is used by Cython. (#1656) @bdice
Hotfix: wrong constant in IVF-PQ fp_8bit2half (#1654) @achirkin
Fix sparse KNN for large batches (#1640) @viclafargue
Fix uploading of RAFT nightly packages (#1638) @dantegd
Fix cagra multi CTA bug (#1628) @enp1s0
pass correct stream to cutlass kernel launch of L2/cosine pairwise distance kernels (#1597) @mdoijade
Fix launchconfig y-gridsize too large in epilogue kernel (#1586) @mfoerste4
Fix update version and pinnings for 23.08. (#1556) @bdice
Fix for function exposing KNN merge (#1418) @viclafargue

📖 Documentation

Critical doc fixes and updates for 23.08 (#1705) @cjnolet
Fix the documentation about changing the logging level (#1596) @enp1s0
Fix raft::bitonic_sort small usage example (#1580) @enp1s0

🚀 New Features

Use rapids-cmake new parallel testing feature (#1623) @robertmaynard
Add support for row-major slice (#1591) @lowener
IVF-PQ tutorial notebook (#1544) @achirkin
[FEA] Masked NN for connect_components (#1445) @tarang-jain
raft: Build CUDA 12 packages (#1388) @vyasr
Limiting workspace memory resource (#1356) @achirkin

🛠️ Improvements

Pin dask and distributed for 23.08 release (#1711) @galipremsagar
Add algo parameter for CAGRA ANN bench (#1687) @tfeher
ANN benchmarks python wrapper for splitting billion-scale dataset groundtruth (#1679) @divyegala
Rename CAGRA parameter num_parents to search_width (#1676) @tfeher
Renaming namespaces to promote CAGRA from experimental (#1666) @cjnolet
CAGRA Python wrappers (#1665) @dantegd
Add notebook for Vector Search - Question Retrieval (#1662) @lowener
Fix CMake CUDA support for pylibraft when raft is found. (#1659) @bdice
Cagra ANN benchmark improvements (#1658) @tfeher
ANN-benchmarks: avoid using the dataset during search when possible (#1657) @achirkin
Revert CUDA 12.0 CI workflows to branch-23.08. (#1652) @bdice
ANN: Optimize host-side refine (#1651) @achirkin
Cagra template instantiations (#1650) @tfeher
Modify comm_split to avoid ucp (#1649) @ChuckHastings
Stop using setup.py in build.sh (#1645) @vyasr
IVF-PQ: Add a (faster) direct conversion fp8->half (#1644) @achirkin
Simplify bench/ann scripts to Python based module (#1642) @divyegala
Further removal of uses-setup-env-vars (#1639) @dantegd
Drop blank line in raft-dask/meta.yaml (#1637) @jakirkham
Enable conservative memory allocations for RAFT IVF-Flat benchmarks. (#1634) @tfeher
[FEA] Codepacking for IVF-flat (#1632) @tarang-jain
Fixing ann bench cmake (and docs) (#1630) @cjnolet
[WIP] Test CI issues (#1626) @VibhuJawa
Set pool memory resource for raft IVF ANN benchmarks (#1625) @tfeher
Adding sort option to matrix::select_k api (#1615) @cjnolet
CAGRA max_queries auto configuration (#1613) @enp1s0
Use exceptions instead of exit(-1) (#1594) @benfred
[REVIEW] Add scheduler_file argument to support MNMG setup (#1593) @VibhuJawa
Rename the CAGRA prune function to optimize (#1588) @enp1s0
This PR adds support to __half and nb_bfloat16 to myAtomicReduce (#1585) @Kh4ster
[IMP] move core CUDA RT macros to cuda_rt_essentials.hpp (#1584) @MatthiasKohl
preprocessor syntax fix (#1582) @AyodeAwe
use rapids-upload-docs script (#1578) @AyodeAwe
Unpin dask and distributed for development and fix merge_labels test (#1574) @galipremsagar
Remove documentation build scripts for Jenkins (#1570) @ajschmidt8
Add support to __half and nv_bfloat16 to most math functions (#1554) @Kh4ster
Add RAFT ANN benchmark for CAGRA (#1552) @enp1s0
Update CAGRA knn_graph_sort to use Raft::bitonic_sort (#1550) @enp1s0
Add identity matrix function (#1548) @lowener
Unpin scikit-build upper bound (#1547) @vyasr
Migrate wheel workflow scripts locally (#1546) @divyegala
Add sample filtering for ivf_flat. Filtering code refactoring and cleanup (#1541) @alexanderguzhva
CAGRA pad dataset for 128bit vectorized load (#1505) @tfeher
Sparse Pairwise Distances API Updates (#1502) @divyegala
Add CAGRA gbench (#1496) @tfeher
Cagra index construction without copying device mdarrays (#1494) @tfeher

v23.10.00a

9 months ago

🔗 Links

🚨 Breaking Changes

Update CAGRA serialization (#1755) @benfred
Improvements to ANN Benchmark Python scripts and docs (#1734) @divyegala
Update to Cython 3.0.0 (#1688) @vyasr
ANN-benchmarks: switch to use gbench (#1661) @achirkin

🐛 Bug Fixes

[BUG] Fix search parameter check in CAGRA (#1784) @enp1s0
IVF-Flat: fix search batching (#1764) @achirkin
Using expanded distance computations in pylibraft (#1759) @cjnolet
Fix ann-bench Documentation (#1754) @divyegala
Make get_cache_idx a weak symbol with dummy template (#1733) @ahendriksen
Fix IVF-PQ fused kernel performance problems (#1726) @achirkin
Fix build.sh to enable NEIGHBORS_ANN_CAGRA_TEST (#1724) @enp1s0
Fix template types for create_descriptor function. (#1680) @csadorf

📖 Documentation

Fix the CAGRA paper citation (#1788) @enp1s0
Add citation info for the CAGRA paper preprint (#1787) @enp1s0
[DOC] Fix grouping for ANN in C++ doxygen (#1782) @lowener
Update RAFT documentation (#1717) @lowener
Additional polishing of README and docs (#1713) @cjnolet

🚀 New Features

Adding config files for remaining (relevant) ann-benchmarks million-scale datasets (#1761) @cjnolet
[FEA] Provide device_resources_manager for easy generation of device_resources (#1716) @wphicks

🛠️ Improvements

Fixing a couple security concerns in raft-dask nccl unique id generation (#1785) @cjnolet
Don't serialize dataset with CAGRA bench (#1781) @benfred
Use copy-pr-bot (#1774) @ajschmidt8
Improvements to raft-ann-bench scripts, docs, and benchmarking implementations. (#1769) @cjnolet
Unpin dask and distributed for 23.10 development (#1760) @galipremsagar
Update CAGRA serialization (#1755) @benfred
Improvements to ANN Benchmark Python scripts and docs (#1734) @divyegala
Fixing forward merger for 23.08 -> 23.10 (#1731) @cjnolet
[FEA] Use CAGRA in C++ template (#1730) @lowener
fixed box around raft image (#1710) @nwstephens
Enable CUTLASS-based distance kernels on CTK 12 (#1702) @ahendriksen
Update bench-ann configuration (#1696) @lowener
Update to Cython 3.0.0 (#1688) @vyasr
Update CMake version (#1677) @vyasr
Branch 23.10 merge 23.08 (#1672) @vyasr
ANN-benchmarks: switch to use gbench (#1661) @achirkin