A unified SQL query interface and portable runtime to locally materialize, accelerate, and query datasets from any database, data warehouse, or data lake.
The v0.12.2-alpha release introduces data streaming and key-pair authentication for the Snowflake data connector, enables general append mode data refreshes for time-series data, improves connectivity error messages, adds nested folders support for the S3 data connector, and exposes nodeSelector and affinity keys in the Helm chart for better Kubernetes management.
Improved Connectivity Error Messages: Error messages provide clearer, actionable guidance for misconfigured settings or unreachable data connectors.
Snowflake Data Connector Improvements: Enables data streaming by default and adds support for key-pair authentication in addition to passwords.
API for Refresh SQL Updates: Update dataset Refresh SQL via API.
Append Data Refresh: Append mode data refreshes for time-series data are now supported for all data connectors. Specify a dataset time_column with refresh_mode: append to only fetch data more recent than the latest local data.
Docker Image Update: The spiceai/spiceai:latest Docker image now includes the ODBC data connector. For a smaller footprint, use spiceai/spiceai:latest-slim.
Helm Chart Improvements: nodeSelector and affinity keys are now supported in the Helm chart for improved Kubernetes deployment management.
POST /v1/datasets/:name/refresh to POST /v1/datasets/:name/acceleration/refresh to be consistent with the Spicepod.yaml structure.release feature in docker image by @ewgenius in https://github.com/spiceai/spiceai/pull/1324
DataConnectorResult and DataConnectorError by @ewgenius in https://github.com/spiceai/spiceai/pull/1339
Full Changelog: https://github.com/spiceai/spiceai/compare/v0.12.1-alpha...v0.12.2-alpha
The v0.12.1-alpha release introduces a new Snowflake data connector, support for UUID and TimestampTZ types in the PostgreSQL connector, and improved error messages across all data connectors. The Clickhouse data connector enables data streaming by default. The public SQL interface now restricts DML and DDL queries. Additionally, accelerated tables now fully support NULL values, and issues with schema conversion in these tables have been resolved.
Snowflake Data Connector: Initial support for Snowflake as a data source.
Clickhouse Data Streaming: Enables data streaming by default, eliminating in-memory result collection.
Read-only SQL Interface: Disables DML (INSERT/UPDATE/DELETE) and DDL (CREATE/ALTER TABLE) queries for improved data source security.
Error Message Improvements: Improved the error messages for commonly encountered issues with data connectors.
Accelerated Tables: Supports NULL values across all data types and fixes schema conversion errors for consistent type handling.
GITHUB_TOKEN environment variable in the installation script, if available, to avoid rate limiting in CI workflows by @ewgenius in https://github.com/spiceai/spiceai/pull/1302
spice login spark by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1303
Full Changelog: https://github.com/spiceai/spiceai/compare/v0.12.0-alpha...v0.12.1-alpha
The v0.12-alpha release introduces Clickhouse and Apache Spark data connectors, adds support for limiting refresh data periods for temporal datasets, and includes upgraded Spice Client SDKs compatible with Spice OSS.
Clickhouse data connector: Use Clickhouse as a data source with the clickhouse: scheme.
Apache Spark Connect data connector: Use Apache Spark Connect connections as a data source using the spark: scheme.
Refresh data window: Limit accelerated dataset data refreshes to the specified window, as a duration from now configuration setting, for faster and more efficient refreshes.
ODBC data connector: Use ODBC connections as a data source using the odbc: scheme. The ODBC data connector is currently optional and not included in default builds. It can be conditionally compiled using the odbc cargo feature when building from source.
Spice Client SDK Support: The official Spice SDKs have been upgraded with support for Spice OSS.
refresh_interval acceleration setting and been changed to refresh_check_interval to make it clearer it is the check versus the data interval.SELECT count(*) for Sqlite Data Accelerator by @sgrebnov in https://github.com/spiceai/spiceai/pull/1166
show tables in Spice SQL & update next version to v0.12.0-alpha by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1206
Full Changelog: https://github.com/spiceai/spiceai/compare/v0.11.1-alpha...v0.12.0-alpha
The v0.11.1-alpha release introduces retention policies for accelerated datasets, native Windows installation support, and integration of catalog and schema settings for the Databricks Spark connector. Several bugs have also been fixed for improved stability.
Retention Policies for Accelerated Datasets: Automatic eviction of data from accelerated time-series datasets when a specified temporal column exceeds the retention period, optimizing resource utilization.
Windows Installation Support: Native Windows installation support, including upgrades.
Databricks Spark Connect Catalog and Schema Settings: Improved translation between DataFusion and Spark, providing better Spark Catalog support.
refresh_sql and manual refresh to e2e tests by @sgrebnov in https://github.com/spiceai/spiceai/pull/1125
spice dataset configure by @ewgenius in https://github.com/spiceai/spiceai/pull/1140
spice upgrade on Windows by @sgrebnov in https://github.com/spiceai/spiceai/pull/1155
Full Changelog: https://github.com/spiceai/spiceai/compare/v0.11.0-alpha...v0.11.1-alpha
The Spice v0.11.0-alpha release significantly improves the Databricks data connector with Databricks Connect (Spark Connect) support, adds the DuckDB data connector, and adds the AWS Secrets Manager secret store. In addition, enhanced control over accelerated dataset refreshes, improved SSL security for MySQL and PostgreSQL connections, and overall stability improvements have been added.
DuckDB data connector: Use DuckDB databases or connections as a data source.
AWS Secrets Manager Secret Store: Use AWS Secrets Managers as a secret store.
Custom Refresh SQL: Specify a custom SQL query for dataset refresh using refresh_sql.
Dataset Refresh API: Trigger a dataset refresh using the new CLI command spice refresh or via API.
Expanded SSL support for Postgres: SSL mode now supports disable, require, prefer, verify-ca, verify-full options with the default mode changed to require. Added pg_sslrootcert parameter for setting a custom root certificate and the pg_insecure parameter is no longer supported.
Databricks Connect: Choose between using Spark Connect or Delta Lake when using the Databricks data connector for improved performance.
Improved SSL support for Postgres: ssl mode now supports disable, require, prefer, verify-ca, verify-full options with default mode changed to require.
Added pg_sslrootcert parameter to allow setting custom root cert for postgres connector, pg_insecure parameter is no longer supported as redundant.
Internal architecture refactor: The internal architecture of spiced was refactored to simplify the creation data components and to improve alignment with DataFusion concepts.
@edmondop's first contribution github.com/spiceai/spiceai/pull/1110!
NULL values by @gloomweaver in https://github.com/spiceai/spiceai/pull/1067
NULL values for NUMERIC by @gloomweaver in https://github.com/spiceai/spiceai/pull/1068
spice refresh CLI command for dataset refresh by @sgrebnov in https://github.com/spiceai/spiceai/pull/1112
TEXT and DECIMAL types support and properly handling NULL for MySQL by @gloomweaver in https://github.com/spiceai/spiceai/pull/1067
DATE and TINYINT types support for MySQL by @ewgenius in https://github.com/spiceai/spiceai/pull/1065
ssl_rootcert_path parameter for MySql data connector by @ewgenius in https://github.com/spiceai/spiceai/pull/1079
LargeUtf8 support and explicitly passing the schema to data accelerator SqlTable by @phillipleblanc in https://github.com/spiceai/spiceai/pull/1077
pg_insecure parameter support from Postgres by @ewgenius in https://github.com/spiceai/spiceai/pull/1081
Full Changelog: https://github.com/spiceai/spiceai/compare/v0.10.2-alpha...v0.11.0-alpha
The v0.10.2-alpha release adds the MySQL data connector and makes external data connections more robust on initialization.
MySQL data connector: Connect to any MySQL server, including SSL support.
Data connections verified at initialization: Verify endpoints and authorization for external data connections (e.g. databricks, spice.ai) at initialization.
show tables; parsing in the Spice SQL repl.lookback_size (& improve SpiceAI's ModelSource) by @Jeadie in https://github.com/spiceai/spiceai/pull/1016
Full Changelog: https://github.com/spiceai/spiceai/compare/v0.10.1-alpha...v0.10.2-alpha
The v0.10.1-alpha release focuses on stability, bug fixes, and usability by improving error messages when using SQLite data accelerators, improving the PostgreSQL support, and adding a basic Helm chart.
Improved PostgreSQL support for Data Connectors TLS is now supported with PostgreSQL Data Connectors and there is improved VARCHAR and BPCHAR conversions through Spice.
Improved Error messages Simplified error messages from Spice when propagating errors from Data Connectors and Accelerator Engines.
Spice Pods Command The spice pods command can give you quick statistics about models, dependencies, and datasets that are loaded by the Spice runtime.
spice login in environments with no browser. (https://github.com/spiceai/spiceai/pull/994)spice pods Returns incorrect counts. (https://github.com/spiceai/spiceai/pull/998)Announcing the release of Spice.ai v0.10-alpha! 🎉
The Spice.ai v0.10-alpha release focused on additions and updates to improve stability, usability, and the overall Spice developer experience.
Public Bucket Support for S3 Data Connector: The S3 Data Connector now supports public buckets in addition to buckets requiring an access id and key.
JDBC-Client Connectivity: Improved connectivity for JDBC clients, like Tableau.
User Experience Improvements:
spice login postgres command, streamlining the process for connecting to PostgreSQL databases.Grafana Dashboard: Improving the ability to monitor Spice deployments, a standard Grafana dashboard is now available.
spice login postgres commandspice status with dataset metricsshow tables outputThe v0.9.1 release focused on stability, bug fixes, and usability by adding spice CLI commands for listing Spicepods (spice pods), Models (spice models), Datasets (spice datasets), and improved status (spice status) details. In addition, the Arrow Flight SQL (flightsql) data connector and SQLite (sqlite) data store were added.
FlightSQL data connector: Arrow Flight SQL can now be used as a connector for federated SQL query.
SQLite data backend: SQLite can now be used as a data store for acceleration.
flightsql).sqlite).spice pods, spice status, spice datasets, and spice models CLI commands.GET /v1/spicepods API for listing loaded Spicepods.spiced Docker CI build and release.linux/arm64 binary build.spice sql REPL panics when query result is too large. (https://github.com/spiceai/spiceai/pull/875)--access-secret in spice s3 login. (https://github.com/spiceai/spiceai/pull/894)The v0.9 release adds several data connectors including the Spice data connector for the ability to connect to other spiced instances. Improved observability for spiced has been added with the new /metrics endpoint for monitoring deployed instances.
Arrow Flight SQL endpoint: The Arrow Flight endpoint now supports Flight SQL, including JDBC, ODBC, and ADBC enabling database clients like DBeaver or BI applications like Tableau to connect to and query the Spice runtime.
Spice.ai data connector: Use other Spice runtime instances as data connectors for federated SQL query across Spice deployments and for chaining Spice runtimes.
Keyring secret store: Use the operating system native credential store, like macOS keychain for storing secrets used by spiced.
PostgreSQL data connector: PostgreSQL can now be used as both a data store for acceleration and as a connector for federated SQL query.
Databricks data connector: Databricks as a connector for federated SQL query across Delta Lake tables.
S3 data connector: S3 as a connector for federated SQL query across Parquet files stored in S3.
Metrics endpoint: Added new /metrics endpoint for spiced observability and monitoring with the following metrics:
- spiced_runtime_http_server_start counter
- spiced_runtime_flight_server_start counter
- datasets_count gauge
- load_dataset summary
- load_secrets summary
- datasets/load_error counter
- datasets/count counter
- models/load_error counter
- models/count counter
keyring).postgres).spiceai).databricks) - Delta Lake support.s3) - Parquet support./v1/models API./v1/status API./metrics API.