TypeDB: the polymorphic database powered by types
Install & Run: https://typedb.com/docs/home/install
Allow disconnected relations
We implement the suggestion from https://github.com/vaticle/typedb/issues/6920, which will allow relations to exist without role players for the duration of a write transaction. At the end of the transaction, any relations without role players are deleted (preserving the existing behaviour of automatically deleting relations without role players, but we delay the operation until commit time).
This change addresses https://github.com/vaticle/typedb/issues/6902, https://github.com/vaticle/typedb/issues/6920,
Implement Fetch query
We implement the language specification for TypeQL "Fetch" queries, finalised and implemented at https://github.com/vaticle/typeql/pull/300.
This change adds a new embedded and network API: QueryManager.fetch(). This API accespts TypeQL Fetch queries, and returns JSON-like structures.
Changes to TypeQL Get queries As part of this change, we have updated the existing terminology around "TypeQL Match" queries to be exclusively called "TypeQL Get" queries. TypeQL Get queries now require a 'get' clause be written at the end of queries. For example:
match
$x isa person;
is no longer a valid query and must be updated to:
match
$x isa person;
get; # alternatively 'get $x;' is more precise
An empty 'get' clause with no variables will return all variables from the 'match' clause, exactly as the previous behaviour which specified no get clause at all.
Modifiers on write queries
We now also allow modifiers on any kind of query, including write queries, that uses a 'match' clause, by adding the desired sort, offset, or limit to the end of your queries. For example, we now allow:
match
$x isa person, has age $a;
delete
$x has $a;
sort $a; limit 10;
Fix traversing edges to role vertices Fixes a bug in resolving edge types when traversing to a role vertex. This could cause missing results in queries involving role variables and relations playing roles in relations.
Fix docker container image workdir to include architecture suffix
Bundle java deps ignoring transitive dependencies without maven coordinates
Recent upgrades to bazel build rules for java, protobuf, and grpc meant that their internal dependencies no longer use Maven coordinates, but are more bazel-native in their build process. However, this means that to 'correctly' bundle TypeDB Server, we also need to include the rules' dependencies into our distribution (such as com_google_protobuf). The issue arises that we also directly use these libraries occasionally, and pull them via Maven. As a result, we end up with duplicates: bazel-build JARs, and maven-sourced JARs inserted into our distribution.
Our workaround for now is to skip all bazel-build JARs that don't also have maven coordinates. This is dangerous since it means we don't find out we have missing JARs in our distributions until booting up the server and testing it. This works only because we have manually verified to include all the 'equivalent' JARs from maven in our distribution instead.
Our reasoning for this is that in TypeDB 3.0 (rust) this issue will no longer be present as most dependencies will be compiled into the final static library.
Update reasoner benchmark to avoid role aliases Updates the IAM reasoner benchmark schema to account for role aliases being disallowed. Relaxes some of the counter upper bounds to accommodate the resulting changes in the plans.
Disallow explicit use of role type aliases
We disallow using role type aliases in labels, only permitting their use when querying for role types in relation constraints. This finalises the implementation of behaviours outlined in https://github.com/vaticle/typeql/issues/274#issuecomment-1766214447.
Given
define
employment sub relation, relates employee;
part-time-employment sub employment;
We now disallow
define
person plays part-time-employment:employee;
As this role type is an alias for employment:employee and therefore generally used incorrectly/confusing.
However, matching is still flexibly allowed:
match
(employee: $x) isa part-time-employment;
Apt deployment depends on default-jre instead of openjdk-11-jre to be compatible with more versions
Update README file
Update the README file.
Install & Run: https://typedb.com/docs/typedb/2.x/installation
-Bundle TypeDB console that uses correct temporary directory to unpack resources
Install & Run: https://typedb.com/docs/typedb/2.x/installation
Optimise storage lookup of IIDs
We optimise reasoner and traversal execution by never looking up IIDs in read transactions - they can trivially be converted into vertices in-memory. This is safe to do since we can verify all IIDs submitted into queries by the user are checked before execution, and all IIDs generated internally by the traversal engine or reasoner are always valid within the read transaction.
This should yield a few percent performance improvement in common reasoner query execution.
Optional config values
We implement a new paradigm for configurations in TypeDB: optional primitive values. The outcome of this is that we have the following three top-level paradigms in our configuration file:
enable: true|false option on the same level.null or ~).Points 2 and 3 derive from point 1 in that they allow us to keep all configurations present, but introduce optionality for either entire blocks or leaf configurations.
Install & Run: https://typedb.com/docs/typedb/2.x/installation
Create separate distributions for each OS + Arch
We create 5 separate distributions of TypeDB, one per platform:
linux-x86_64
linux-arm64
mac-x86_64
mac-arm64
windows-x86_64
Please be aware that this means TypeDB distributions are no longer portable between Intel and Mac variants of the same system - eg. from an Intel mac to an ARM mac.
Handle transitivity in [Thing]TypeGetSubtypes requests
We previously ignored the transitivity field in the incoming *TypeGetSubtypes() requests, which led to *TypeGetSubtypes(EXPLICIT) returning all transitive subtypes. This behaviour is now fixed.
Fix APT snapshot test, bazel install on Mac CI
Downgrade CircleCI executors to ensure backwards compatibility
Fix CircleCI assembly test executor types
Update apt assembly to set empty directory permissions explicitly
Replace 'client' terminology with 'driver' terminology
We replace the term 'cluster' with 'enterprise', to reflect the new consistent terminology used through Vaticle. We also replace 'client' with 'driver', where appropriate.
Deploy x86-64 and arm64 apt distributions
We deploy two apt distributions: x86_64 (aka amd64 in apt) and arm64. Apt automatically selects the correct architecture matching the current platform, so users just need to add the apt source and install the server.
Simplify Concept API & unify gRPC API with Cluster (#6761)
We simplify the Concept interface by unifying methods that were previously split into their general and Explicit variants. This allows us to simplify the gRPC protocol and deduplicate the implementation of the methods.
In addition, we adjust the way TypeDB server responds to database and server manager requests to align it with TypeDB Cluster.
Reduce size of looping integration tests
Include console artifact that includes both arm and intel architectures for mac and linux
Delete sonarcloud code analysis job
Replace occurrences of Cluster with Enterprise and Client with Driver, where appropriate
Install & Run: https://typedb.com/docs/typedb/2.x/installation
Create separate distributions for each OS + Arch
We create 5 separate distributions of TypeDB, one per platform:
linux-x86_64
linux-arm64
mac-x86_64
mac-arm64
windows-x86_64
Please be aware that this means TypeDB distributions are no longer portable between Intel and Mac variants of the same system - eg. from an Intel mac to an ARM mac.
Handle transitivity in [Thing]TypeGetSubtypes requests
We previously ignored the transitivity field in the incoming *TypeGetSubtypes() requests, which led to *TypeGetSubtypes(EXPLICIT) returning all transitive subtypes. This behaviour is now fixed.
Fix APT snapshot test, bazel install on Mac CI
Downgrade CircleCI executors to ensure backwards compatibility
Fix CircleCI assembly test executor types
Update apt assembly to set empty directory permissions explicitly
Replace 'client' terminology with 'driver' terminology
We replace the term 'cluster' with 'enterprise', to reflect the new consistent terminology used through Vaticle. We also replace 'client' with 'driver', where appropriate.
Deploy x86-64 and arm64 apt distributions
We deploy two apt distributions: x86_64 (aka amd64 in apt) and arm64. Apt automatically selects the correct architecture matching the current platform, so users just need to add the apt source and install the server.
Simplify Concept API & unify gRPC API with Cluster (#6761)
We simplify the Concept interface by unifying methods that were previously split into their general and Explicit variants. This allows us to simplify the gRPC protocol and deduplicate the implementation of the methods.
In addition, we adjust the way TypeDB server responds to database and server manager requests to align it with TypeDB Cluster.
Reduce size of looping integration tests
Include console artifact that includes both arm and intel architectures for mac and linux
Delete sonarcloud code analysis job
Replace occurrences of Cluster with Enterprise and Client with Driver, where appropriate
Install & Run: https://typedb.com/docs/typedb/2.x/installation
Architectural changes for the new Auditing feature in TypeDB Enterprise Refactors configuration & grpc service classes to allow extending and auditing in TypeDB Enterprise.
Upgrade SpeeDB to 2.6.0 We upgrade to SpeeDB 2.6.0, which even without further configuration improves read performance by 3-5% under load.
Install & Run: https://typedb.com/docs/typedb/2.x/installation
Allow type variables in insert and delete clauses
We allow using type variables in delete and insert clauses. This allows much more expressivity when doing deletes, in particular allowing disjunctions over types in the match, which are then operated over in turn in the delete clause to clean up relations or attributes.
This change resolves https://github.com/vaticle/typedb/issues/6755
Deletes are idempotent instead of throwing
We move partially towards the 'assert-style' deletion outlined in #6882 to allow idempotent deletion. Previously, deletes would throw when trying to delete a previously-removed/non-existent connection.
We now guarantee that all deletes will continue even if trying to delete a relationship, ownership, role player, or concept that doesn't exist (ie. was previously deleted).
For example, the following is now valid:
match
$x has $a, has $b; $b = $a;
delete
$x has $a;
Because the same attribute would be generated twice by the match, the second delete would fail and throw an error. This is now an idempotent operation, which will continue without an error.
We also introduce a new operation to disallow a new class of semantically unsafe queries. We add an explicit 'runtime' type check before performing the idempotent delete operation. This means the following semantically invalid query will still throw an exception when any data matches:
match
$x isa person;
$id isa company-id;
delete
$x has $id;
Role types are always concrete and inheritable
We address https://github.com/vaticle/typeql/issues/203 by making all role types in relations to be concrete - even when the relation type is abstract. The resulting behaviour is exactly as when using non-abstract relation type hierarchies.
The following is now permissible:
define
membership sub relation, abstract, relates member;
group-membership sub membership; // inherit 'member'
<commit>
insert
(member: $x) isa group-membership;
Migration: new abstract relation types defined will automatically have concrete relation roles. However, old databases with pre-existing abstract relation types will preserve role abstractness. The simple way to migrate to the new behaviour is to unset and set the abstractness of each abstract relation type via the Concept API:
# pseudocode
for rel_type in relation_types:
if rel_type.is_abstract():
rel_type.unset_abstract()
rel_type.set_abstract()
commit()
Warning: there is a gap in the semantics of inheriting role types. Role types are considered independent types, and validated as such. In the above example, the role type group-membership:member doesn't actually exist, it aliases to membership:member.
This means, if we write the following:
define
person plays group-membership:member;
this actually implies person plays membership:member.
Further, any relation types that inherit the member role type are automatically playable by the person:
define
person sub entity, plays membership:member;
team-membership sub membership;
<commit>
insert
(member: $x) isa team-membership;
(member: $x) isa group-membership;
(member: $x) isa membership;
These are all legal inserts.
This gap in the expressivity has always existed in the language when using non-abstract relation type hierarchies. Resolving this would imply solving https://github.com/vaticle/typeql/issues/274, either by automatically creating a new member sub-role for each relation sub-type, or by requiring the user do so explicitly. In the meantime, the workaround is to explicitly override the inherited role with a new name in the child:
define
team-membership sub membership, relates team-member as member;
Add configurable reasoner plan logging & periodic printing of counters Adds logging of reasoner plans and reasoner performance counters. These can be toggled through the config file & log-level of respective loggers.
To enable & log reasoner plans:
log:
...
logger:
...
reasoner-planner:
filter: com.vaticle.typedb.core.reasoner.planner
level: debug
output: [ stdout, file ]
To enable & log reasoner counters:
log:
...
logger:
...
reasoner-perf-counters:
filter: com.vaticle.typedb.core.reasoner.common.ReasonerPerfCounters
level: debug
output: [ stdout ]
...
debugger:
reasoner-tracer:
enable: false
type: reasoner-tracer
output: file
reasoner-perf-counters:
enable: true
type: reasoner-perf-counters
Update readme file with the website content
Update the readme file to be in-line with the new typedb.com website.
Install & Run: https://typedb.com/docs/typedb/2.x/installation
Shrink distributions by excluding unnecessary native jars
We discover that our distributions were larger than required, since they included every platform's native or-tools jars. We now manually exclude native or-tools from maven, and include them per-platform individually. This should reduce artifact size by 30-40 MB.
Add logic to allow unlimited archive age or sizes
The change in #6864 enabled configuring log archive retention policies by age or time, but would actually fail if trying to set unlimited time and size limits. The user can now set age limits or aggregate size limits by using the value 0 for either archive-age-limit or archives-size-limit configurations respectively.
Install & Run: https://typedb.com/docs/typedb/2.x/installation
Configure logging to respect size and time limits
TypeDB's logger previously could over-run its expected archives size cap. This occurred because the logging framework expected to have both the time and size window configured for either to work correctly.
TypeDB logger file configuration now has an updated set of options to control both archive retention policies for size and age.
log:
output:
file:
type: file
base-dir: server/logs // renamed from 'directory'
file-size-limit: 50mb // renamed from 'file-size-cap'
archive-grouping: month // new option, using: minute(s) | hour(s) | day(s) | week(s) | month(s) | year(s)
archive-age-limit: 1 year // new option, using: <N> minute(s) | hour(s) | day(s) | week(s) | month(s) | year(s)
archives-size-limit: 1gb // renamed from 'archives-size-cap'
The archive-grouping option configures the rollover and naming policy of archives produced by the logger.
The archive-age-limit option configures how long each archive files are kept. Note that old archives are only deleted when new ones are produced.
The execution semantics are that every period of time defined by 'grouping', files exceeding the time limit are asynchronously deleted, and then oldest files are asynchronously deleted until the total size cap is respected.
In the above example, log files would be compacted into archives monthly, with naming pattern like:
typedb_202306.0.log.gz
typedb_202307.0.log.gz
typedb_202308.0.log.gz
...
Where 1 year's worth of log archives are retained.
Resolves #6854.
Disable stamping to fix windows builds
As of #6853 we include an updated version of rules_jvm_external, which is used to transform maven dependencies into valid Bazel targets. However, the upgrade to version 5 also included a new default feature called 'stamping' (https://bazel.build/docs/user-manual#workspace-status), which works in general but breaks builds on Windows.
This PR disables stamping by default (which supposedly also improves cache hits on remote caches), as well as shortens the Windows build directory to avoid potential long path issues in the future.
Fix NullPointerExceptions during reasoner stream graph construction Fix a concurrency bug leading to rare NullPointerExceptions during the construction of the reasoner graph.
Fix server hang-ups during shutdown
Under exceptional circumstances, such as when the server runs out of memory, the server could fail to shut down. We modify the server shutdown process and unexpected exception handler to shutdown in several stages, in the most severe case halting the JVM runtime immediately.
Made datetime attribute insert and match time-zone invariant
Enables the BDD tests to check timezone-invariance of inserting and reading datetime attributes.
Bump vaticle dependencies for jNaCl Bump vaticle/dependencies to one which includes jNaCL. This is needed for tests in TypeDB enterprise.
Bump dependencies for typedb-enterprise encryption improvements Bump netty dependencies for typedb-enterprise encryption changes.
Merge for 2.19.1 release
Merge for 2.19.1 release.
Install & Run: http://docs.vaticle.com/docs/running-typedb/install-and-run
Fix server hang-ups during shutdown
Under exceptional circumstances, such as when the server runs out of memory, the server could fail to shut down. We modify the server shutdown process and unexpected exception handler to shutdown in several stages, in the most severe case halting the JVM runtime immediately.
Made datetime attribute insert and match time-zone invariant
Enables the BDD tests to check timezone-invariance of inserting and reading datetime attributes.
Update SpeeDB to 2.4.1.5
This version of SpeeDB lowers the dependency version of GLIBC.