TypeDB: the polymorphic database powered by types
Install & Run: https://typedb.com/docs/home/install
Download from TypeDB Package Repository:
Server only: Distributions for 2.28.0
Server + Console: Distributions for 2.28.0
Use Rosetta for Intel Mac jobs in Circle CI
CircleCI is sunsetting its MacOS Intel architecture executors. This PR transitions our Mac x86_64 CI tests to run on ARM executors using Rosetta.
Replace licenses with MPL version 2.0
The release of TypeDB 2.28.0 marks the beginning of the Mozilla Public License (MPL) in the TypeDB stack. MPL is a simpler and more permissive license than AGPL - it encourages sharing of modifications of the source code, while allowing unmodified use in open or proprietary work with minimal restriction. Mozilla's FAQ on the license is an easy entrance on the license.
Fix application of query bounds through match-fetch queries
We fix a bug that was revealed when using match-fetch queries with subqueries that had nested patterns. This change now correctly applies 1) filtering to a parent match query so that the right outputs are generated 2) passes the bounds received from a preceding query correctly into all child patterns of subsequent query clauses.
Check supertypes of relation when inferring player role types during insert
Given the following minimal schema:
define
player sub entity, plays super-relation:super-role;
super-relation sub relation, relates super-role;
sub-relation sub super-relation;
It was impossible to insert an instance of sub-relation relating a player without explicitly specifying its role:
> match $player isa player;
insert ($player) isa sub-relation;
[THW27] Invalid Thing Write: Unable to add role player '$player' to the relation,
as there is no provided or inferrable role type.
The reason for this was that during the handling of the insert query, we check if it can play a role defined directly on the requested relation type, without considering any role types it may have inherited. This PR resolves that issue.
Disable using type information from IID in type inference
We decide to ignore IID information when performing type inference. This comes from the idea that IIDs are data, and the user may be trying to check whether the data conforms to a specific part of the schema. Previously, this will cause a query to throw an error due to type checking incompatibility, which made common expressions in fetch subqueries very difficult
Fetching stocks for books that are able to have stock (some subtypes of books do not have stock, such as 'ebook'):
match
$x isa book;
fetch:
stock: {
match $x isa! $t; $t owns stock;
fetch $x: stock;
};
Which will retrieve stock for books, if the polymorphically selected book instance is able to own 'stock' attributes.
This expression now returns an empty list for instances of books that cannot own stocks instead of returning an error.
Update protocol dependency
Update bat file license headers
Update windows choco dependencies
Sentry transaction: we reintroduce a simple notification task, when diagnostics is enabled, which notifies the diagnostics server of the server ID.
Install & Run: https://typedb.com/docs/home/install
Download from TypeDB Package Repository:
Server only: Distributions for 2.28.0-rc0
Server + Console: Distributions for 2.28.0-rc0
Check supertypes of relation when inferring player role types during insert
Given the following minimal schema:
define
player sub entity, plays super-relation:super-role;
super-relation sub relation, relates super-role;
sub-relation sub super-relation;
It was impossible to insert an instance of sub-relation relating a player without explicitly specifying its role:
> match $player isa player;
insert ($player) isa sub-relation;
[THW27] Invalid Thing Write: Unable to add role player '$player' to the relation,
as there is no provided or inferrable role type.
The reason for this was that during the handling of the insert query, we check if it can play a role defined directly on the requested relation type, without considering any role types it may have inherited. This PR resolves that issue.
Disable using type information from IID in type inference
We decide to ignore IID information when performing type inference. This comes from the idea that IIDs are data, and the user may be trying to check whether the data conforms to a specific part of the schema. Previously, this will cause a query to throw an error due to type checking incompatibility, which made common expressions in fetch subqueries very difficult
Fetching stocks for books that are able to have stock (some subtypes of books do not have stock, such as 'ebook'):
match
$x isa book;
fetch:
stock: {
match $x isa! $t; $t owns stock;
fetch $x: stock;
};
Which will retrieve stock for books, if the polymorphically selected book instance is able to own 'stock' attributes.
This expression now returns an empty list for instances of books that cannot own stocks instead of returning an error.
Update README.md
Update protocol dependency
Update bat file license headers
Replace licenses with MPL version 2.0
Update windows choco dependencies
Sentry transaction
We reintroduce a simple notification task, when diagnostics is enabled, which notifies the diagnostics server of the server ID.
Install & Run: https://typedb.com/docs/home/install
Download from TypeDB Package Repository:
Server only: Distributions for 2.27.0
Server + Console: Distributions for 2.27.0
Metrics and diagnostics service
We introduce a metrics and diagnostics service. The web endpoint is bound at port 4104 by default and exposes metrics formatted for prometheus (http://localhost:4104/metrics?format=prometheus) or as JSON (http://localhost:4104/metrics?format=JSON).
Example metrics:
# TypeDB version: TypeDB Core 2.26.6
# Time zone: Europe/London
# Java version: Azul Systems, Inc. 11.0.15
# Platform: Mac OS X aarch64 13.6.3
# TYPE typedb_attempted_requests_total counter
typedb_attempted_requests_total{kind="CONNECTION_OPEN"} 625
typedb_attempted_requests_total{kind="SERVERS_ALL"} 625
typedb_attempted_requests_total{kind="USER_MANAGEMENT"} 0
typedb_attempted_requests_total{kind="USER"} 0
typedb_attempted_requests_total{kind="DATABASE_MANAGEMENT"} 2606
typedb_attempted_requests_total{kind="DATABASE"} 252
typedb_attempted_requests_total{kind="SESSION"} 1207
typedb_attempted_requests_total{kind="TRANSACTION"} 2835
# TYPE typedb_successful_requests_total counter
typedb_successful_requests_total{kind="CONNECTION_OPEN"} 625
typedb_successful_requests_total{kind="SERVERS_ALL"} 625
typedb_successful_requests_total{kind="USER_MANAGEMENT"} 0
typedb_successful_requests_total{kind="USER"} 0
typedb_successful_requests_total{kind="DATABASE_MANAGEMENT"} 2606
typedb_successful_requests_total{kind="DATABASE"} 252
typedb_successful_requests_total{kind="SESSION"} 1207
typedb_successful_requests_total{kind="TRANSACTION"} 2755
# TYPE typedb_current_count gauge
typedb_current_count{kind="DATABASES"} 2
typedb_current_count{kind="SESSIONS"} 1
typedb_current_count{kind="TRANSACTIONS"} 1
typedb_current_count{kind="USERS"} 0
# TYPE typedb_error_total counter
typedb_error_total{code="TXN23"} 2
typedb_error_total{code="QRY22"} 4
typedb_error_total{code="TXN19"} 2
typedb_error_total{code="THR10"} 19
We also extract the metrics reporter into a separate module within the diagnostics package and redirect it to the dedicated diagnostics service.
Redesign schema modification capabilities
We redesign schema modification to allow much more flexible in-place changes to the database schema. We relax various schema invariants within a schema write transaction, to allow moving and editing schema types on the fly. However, the data is validated against the schema consistency at each step, allowing full and safe use of TypeDB's existing Concept and Query API. Before committing, we can restore schema invariants guided by TypeDB's exceptions API (ConceptManager.getSchemaExceptions()).
This change facilitates large-scale database schema migration. We expect the following workflow to be adopted:
... as TYPE) are allowed to refer to types that are not overridable at that place in the schema. This is common when moving a type from one supertype to a different supertype.
b. Redeclarations are allowed: Declarations of owns, plays, or annotations, may be duplicated in child types. This facilitates moving types from one supertype to a different supertype, or moving declarations up or down the type hierarchy.
c. Relaxed abstract ownership: Types may own abstract attribute types without themselves being abstract.ConceptManager.getSchemaExceptions() (transaction.concepts().getSchemaExceptions() in most drivers).Operations that expand the schema capabilities:
plays or owns
Operations that restrict the schema capabilities:
plays or owns
Disable monitoring in assembly tests
Failure to bind a port for the monitoring server for some reason results in a failure of assembly tests on Windows. Disabling monitoring should resolve the issue.
Fix Owns constraint toString()
Update anonymous variable in delete error message
Merge release branch back to development after 2.27.0-rc0
We merge changes in the master branch back to development.
Update docs/banner.png for the README file
Update the banner image in the README file.
Update maven artifacts
Update typedb-console-runner version
Clarify error message on illegal type specifier in query
Rename TypeDB Enterprise error to TypeDB Cloud
Whenever an error message previously referred to TypeDB "Enterprise" edition, it now refers to "Cloud" instead.
Install & Run: https://typedb.com/docs/home/install
Download from TypeDB Package Repository:
Server only: Distributions for 2.27.0-rc0
Server + Console: Distributions for 2.27.0-rc0
Metrics and diagnostics service
We introduce a metrics and diagnostics service. The web endpoint is bound at port 4104 by default and exposes metrics formatted for prometheus (http://localhost:4104/metrics?format=prometheus) or as JSON (http://localhost:4104/metrics?format=JSON).
Example metrics:
# TypeDB version: TypeDB Core 2.26.6
# Time zone: Europe/London
# Java version: Azul Systems, Inc. 11.0.15
# Platform: Mac OS X aarch64 13.6.3
# TYPE typedb_attempted_requests_total counter
typedb_attempted_requests_total{kind="CONNECTION_OPEN"} 625
typedb_attempted_requests_total{kind="SERVERS_ALL"} 625
typedb_attempted_requests_total{kind="USER_MANAGEMENT"} 0
typedb_attempted_requests_total{kind="USER"} 0
typedb_attempted_requests_total{kind="DATABASE_MANAGEMENT"} 2606
typedb_attempted_requests_total{kind="DATABASE"} 252
typedb_attempted_requests_total{kind="SESSION"} 1207
typedb_attempted_requests_total{kind="TRANSACTION"} 2835
# TYPE typedb_successful_requests_total counter
typedb_successful_requests_total{kind="CONNECTION_OPEN"} 625
typedb_successful_requests_total{kind="SERVERS_ALL"} 625
typedb_successful_requests_total{kind="USER_MANAGEMENT"} 0
typedb_successful_requests_total{kind="USER"} 0
typedb_successful_requests_total{kind="DATABASE_MANAGEMENT"} 2606
typedb_successful_requests_total{kind="DATABASE"} 252
typedb_successful_requests_total{kind="SESSION"} 1207
typedb_successful_requests_total{kind="TRANSACTION"} 2755
# TYPE typedb_current_count gauge
typedb_current_count{kind="DATABASES"} 2
typedb_current_count{kind="SESSIONS"} 1
typedb_current_count{kind="TRANSACTIONS"} 1
typedb_current_count{kind="USERS"} 0
# TYPE typedb_error_total counter
typedb_error_total{code="TXN23"} 2
typedb_error_total{code="QRY22"} 4
typedb_error_total{code="TXN19"} 2
typedb_error_total{code="THR10"} 19
We also extract the metrics reporter into a separate module within the diagnostics package and redirect it to the dedicated diagnostics service.
Redesign schema modification capabilities
We redesign schema modification to allow much more flexible in-place changes to the database schema. We relax various schema invariants within a schema write transaction, to allow moving and editing schema types on the fly. However, the data is validated against the schema consistency at each step, allowing full and safe use of TypeDB's existing Concept and Query API. Before committing, we can restore schema invariants guided by TypeDB's exceptions API (ConceptManager.getSchemaExceptions()).
Clarify error message on illegal type specifier in query
Rename TypeDB Enterprise to TypeDB Cloud
Whenever an error message previously referred to TypeDB "Enterprise" edition, it now refers to "Cloud" instead.
Install & Run: https://typedb.com/docs/home/install
Download from TypeDB Package Repository:
Server only: Distributions for 2.26.6
Server + Console: Distributions for 2.26.6
Enable Unicode variable names
We update to TypeQL with Unicode support in both value and concept variables. This makes the following valid TypeQL:
match $人 isa person, has name "Liu"; get $人;
match $אדם isa person, has name "Solomon"; get $אדם;
This change is fully backwards compatible.
Fix schema export of overridden plays edge Make schema exports consistent with the TypeQL spec of overridden roles in 'plays' declarations being unscoped labels.
Fix bug when overriding committed role-types with newly defined role-types Fixes a bug which creates an invalid override when a committed role-type is overridden with a newly defined role-type.
Fix commit-time cleanup of chains of empty relations
In cases where chains of empty relations existed, for example r1 -> r2 -> r3... -> rX, it was possible that TypeDB did not correctly execute the automatic relation cleanup due to the order the relations happened to be traversed.
TypeDB now retries the commit-time cleanup of empty relations until there are no new deletions. We accept the higher runtime cost of this operation since 1) we expect the number of modified relations in a transaction to be relatively small (no more than thousands) 2) relation chains are relatively rare 3) relation chains that must clean up in dependent fashion are extremely rare.
Fix apt deployment: correct entry point script
When assembling a server + console bundle for apt deployment, the console entry point would sometimes override the bundle entry point, making the server inaccessible. We resolve that issue by explicitly filtering out the console entry point during assembly.
Generate persistent server ID
We update the server ID to be generated once, then persisted into the server's data directory. A fallback ID of of _0 is also used if this fails for any reason.
Transition from standalone typedb-common to typeql/common
We update Bazel dependencies and target paths following the merging of typedb-common into vaticle/typeql (see https://github.com/vaticle/typeql/pull/313).
Migrate artifact hosting to cloudsmith Updates artifact credentials, and deployment & consumption rules to use cloudsmith (repo.typedb.com) instead of the self-hosted sonatype repository (repo.vaticle.com).
Improve error message when using anonymous variable in delete query
We now throw an explicit error message when a user tries to delete a 'thing' variable that is not named. For example:
match $x isa person;
delete $x has name 'John';
Will now explicitly throw an exception saying that the 'delete' variables must be matched first.
Disable diagnostics in Docker and Assembly tests
We disable diagnostics in Docker and Assembly tests. This also required allow the TypeDB Runner library to accept and transmit arguments to the binary it boots.
Ingest binary and test-runner package from common
We bring in the binary package and test-runner from typedb-common. typedb-runner is deployed to maven such that we can safely depend on it from other repos without creating Bazel dependency cycles.
Improve fatal error handling and shutdown
We streamline the error handling infrastructure by requiring all TypeDBExceptions have an associated error code, avoiding simple wrappers of Java or Storage exceptions without any context. We also ensure that all errors originating in TypeDB, such as TypeQL parser errors, are wrapped in TypeDBExceptions.
The result is the following invariant:
Additionally, Diagnostics are now implemented via a Singleton in order to inject different error reporting for TypeDB Core or Cloud. This means that all tests must now initialise a Diagnostics singleton to no-op via Diagnostics.initialiseNoop() before executing.
Pin CircleCI Ubuntu image to a specifici Ubuntu 20.04 release
Explicitly install python tool dependencies
Since the upgrade to rules-python v0.24 (https://github.com/vaticle/dependencies/pull/460), we are required to explicitly install python dependencies in the WORKSPACE file. The python tools happened to be unused, so these errors were not visible until the sync dependencies tool was restored.
Sync dependencies in CI
We add a sync-dependencies job to be run in CI after successful snapshot and release deployments. The job sends a request to vaticle-bot to update all downstream dependencies.
Note: this PR does not update the dependencies repo dependency. It will be updated automatically by the bot during its first pass.
//ci:sync-dependencies: Set up CI filters for master-development workflow
Catch any exceptions during error diagnostics configuration
Diagnostics can be re-initialised to facilitate in-memory tests
Disable diagnostics in Reasoner benchmarks and Reasoner behaviour tests
Fix negated traversals so filters apply after the negation
Fixes a bug where the get clause filtered out concepts before the negation was applied. This lead to the negation being evaluated without that variable being bound.
Shorten diagnostic ID to 16 hex chars
Install & Run: https://typedb.com/docs/home/install
Download from TypeDB Package Repository:
Server only: Distributions for 2.26.6-rc1
Server + Console: Distributions for 2.26.6-rc1
Enable Unicode variable names
We update to TypeQL with Unicode support in both value and concept variables. This makes the following valid TypeQL:
match $人 isa person, has name "Liu"; get $人;
match $אדם isa person, has name "Solomon"; get $אדם;
This change is fully backwards compatible.
Generate persistent server ID
We update the server ID to be generated once, then persisted into the server's data directory. A fallback ID of of _0 is also used if this fails for any reason.
Transition from standalone typedb-common to typeql/common
We update Bazel dependencies and target paths following the merging of typedb-common into vaticle/typeql (see https://github.com/vaticle/typeql/pull/313).
Migrate artifact hosting to cloudsmith Updates artifact credentials, and deployment & consumption rules to use cloudsmith (repo.typedb.com) instead of the self-hosted sonatype repository (repo.vaticle.com).
Improve error message when using anonymous variable in delete query
We now throw an explicit error message when a user tries to delete a 'thing' variable that is not named. For example:
match $x isa person;
delete $x has name 'John';
Will now explicitly throw an exception saying that the 'delete' variables must be matched first.
Disable diagnostics in Docker and Assembly tests
We disable diagnostics in Docker and Assembly tests. This also required allow the TypeDB Runner library to accept and transmit arguments to the binary it boots.
Ingest binary and test-runner package from common
We bring in the binary package and test-runner from typedb-common. typedb-runner is deployed to maven such that we can safely depend on it from other repos without creating Bazel dependency cycles.
Improve fatal error handling and shutdown
We streamline the error handling infrastructure by requiring all TypeDBExceptions have an associated error code, avoiding simple wrappers of Java or Storage exceptions without any context. We also ensure that all errors originating in TypeDB, such as TypeQL parser errors, are wrapped in TypeDBExceptions.
The result is the following invariant:
Additionally, Diagnostics are now implemented via a Singleton in order to inject different error reporting for TypeDB Core or Cloud. This means that all tests must now initialise a Diagnostics singleton to no-op via Diagnostics.initialiseNoop() before executing.
Explicitly install python tool dependencies
Since the upgrade to rules-python v0.24 (https://github.com/vaticle/dependencies/pull/460), we are required to explicitly install python dependencies in the WORKSPACE file. The python tools happened to be unused, so these errors were not visible until the sync dependencies tool was restored.
Sync dependencies in CI
We add a sync-dependencies job to be run in CI after successful snapshot and release deployments. The job sends a request to vaticle-bot to update all downstream dependencies.
Note: this PR does not update the dependencies repo dependency. It will be updated automatically by the bot during its first pass.
//ci:sync-dependencies: Set up CI filters for master-development workflow
Catch any exceptions during error diagnostics configuration
Diagnostics can be re-initialised to facilitate in-memory tests
Disable diagnostics in Reasoner benchmarks and Reasoner behaviour tests
Fix negated traversals so filters apply after the negation
Fixes a bug where the get clause filtered out concepts before the negation was applied. This lead to the negation being evaluated without that variable being bound.
Shorten diagnostic ID to 16 hex chars
Install & Run: https://typedb.com/docs/home/install
Fix fetch sub-query aggregation null pointer
We fix a bug where a Fetch query with a Get-Aggregate subquery that returned an empty (ie. undefined) answer throw a null pointer exception.
For example this used to error if 'Alice' doesn't have any salaries in the database, since a 'sum' is undefined for 0 entries.
match
$x isa person, has name $n; $n == "Alice";
fetch
$n as "name";
total-salary: {
match $x has salary $s;
get $s;
sum $s;
};
We now return the following JSON structure
[
{
"name": {"value": "Alice", "type": {"label": "name", "root": "attribute", "value_type": "string"}},
"total-salary": null
}
]
Implement server diagnostics
We implement completely anonymous diagnostics reporting in TypeDB Core using Sentry. This includes sending unexpected system errors and expected user errors to the backend reporting system. We also submit occasional system status updates for number of databases/database size (daily) and periodically profile a transaction. This instrumentation should help get real-world usage and error analysis to help guide the development and efforts in TypeDB.
Error and diagnostic reporting can be completely disabled in the configuration file by setting:
diagnostics:
reporting:
enable: false
or booting the server with --diagnostics.reporting.enable=false.
Install & Run: https://typedb.com/docs/home/install
This version is backwards compatible at the network and data-storage layers.
The console connection command for TypeDB command has changed. Please use:
typedb console --core=<uri>
instead of
typedb console --server=<uri>
Note that leaving out the argument
--core=<uri>will default to--core= localhost:1729.
Update dependencies: rules_rust 0.31, updated typedb-common runners
We update dependencies to make the changes transitively available to TypeDB Cloud.
Fetch query can infer fetched attributes
We fix #6952 , by allowing attribute fetching to trigger reasoning. For example;
Given:
define
rule has-name: when { $x isa person; } then { $x has name "John"; };
Then the query:
match $x isa person;
fetch $x: name;
Can now retrieve name "John" as part of the fetch response.
Update assembly tests to use new console flags
Update assembly tests to use new --core=<address> flag in place of the old --server=<address>.
Null checks in TypeService
We expand null checking in transaction TypeService, which is used for handling Concept API type requests. This replaces the uninformative NPE with a Missing Concept error message.
Update Console artifact: Cloud connection rename
We update the console artifact to make it transitively available to the TypeDB Cloud distribution. The principal change in Console is the updated CLI flag to connect to a Cloud instance (--cloud rather than --enterprise).
Update entry point runner for better UX
We update to the updated entry point for the assembled TypeDB Core, which omits console from the usage help if it is not present to avoid confusion. See https://github.com/vaticle/typedb/issues/6942 for more details.
The command to boot up the server (unchanged):
typedb server --server.address=<address>
The command to boot up the console:
typedb console --core=<address>
We also improve the UX of the windows version of the entry point. Console no longer opens in a new window, --help is printed inline as expected, and in the event of the server failure, the logs are displayed to the user. See https://github.com/vaticle/typedb-common/pull/158 for more details.
Fetch output format update: attribute type includes "value_type"
We update the serializiation schema of TypeQL Fetch query outputs: attribute type serialization now includes its value_type. This change makes the output symmetric between raw values and attributes:
{
"raw_value": { "value": "...", "value_type": "string" },
"attribute": { "value": "...", "type": { "label": "T", "value_type": "string", "root": "attribute" } }
}
Update README.md
Update README.md
Update README.md
Update README.md
Update the readme file: fix TypeQL link
Update the readme to fix the TypeQL link.
Fix broken forum badge
The goal of this PR is to address a broken forum badge caused by a change in Discourse's statistics endpoint.
Install & Run: https://typedb.com/docs/home/install
Bundle TypeDB Console with improved error messages
Ignore data directories that aren't TypeDB databases
TypeDB ignores directories in its configured 'data' directory that do not contain the subdirectories 'data' and 'schema'. In the past, the any directory was loaded as a TypeDB database, which could cause the server to crash on bootup.
This should help with OS-created directories in some system (such as Lost+Found), or users' debugging directories.
Remove sonarcloud dependencies
Simplify Github templates
Install & Run: https://typedb.com/docs/home/install
Allow directory to be created on bootup
We allow the data directory to be created on bootup, resolving https://github.com/vaticle/typedb/issues/6919
Allow parallelised concept and manager APIs
We implement https://github.com/vaticle/typedb/issues/6922, by allowing parallelised read queries for all protocol APIs that can be classified as read-only requests.
Assemble and deploy a single apt package
We no longer deploy typedb-server or typedb-all as a separate package, but rather a single package called typedb which contains server, console, and the binary. This approach now mirrors the 'brew' installation package.
Update readme - Add a link to the 25 queries page
Add a link to the TypeDB in 25 queries page to the readme file.