Apollo Delphinius

The Apollo Delphinius project is an experimental multi-tenant, distributed system platform. Apollo provides a secure communications overlay using Fireflies. The consensus layer is supplied by an asynchronous bft consensus protocol. The sql state interface is via a JDBC connection over replicated SQL state machines, supported by checkpointed CHOAM linear logs. Decentralized identity and key management are provided as a foundational service and integrated into the MTLS grpc communication.

The target service goal is a multi-tenant secrets manager that provides a wide area replicated, low latency service for managing identity, key management, access control and verifiable credentials.

Build Status

The Java Maven CI is now integrated, and given how weak these CI daemons are, this should guarantee reproducible clean builds from the command line maven.

Not A Coin Platform™

Apollo is not designed for coins, rather it is a distributed multi-tenant database. While the systems and mechanisms of Apollo can be used for coins/etc, the design goals are much different. Thus, no coins for you.

Some Features

• Multi tenant isolation enclaves using GraalVM Isolates
• Self-contained cryptography module — Self describing Digests, Signatures and Identifiers, solid Bloom Filters, windows, etc
• Decentralized Identifier-based foundation and key management infrastructure, based on the Key Event Receipt Infrastructure (KERI)
• Secure and trusted attestation, identity bootstrapping and secrets provisioning
• MTLS network communication — KERI for MTLS certificate authentication. Local communication simulation for simplified multi-node simulation for single process (IDE) testing
• Multi instance GRPC service routing - Context keyed services and routing framework
• Byzantine intrusion tolerant secure membership and communications overlay providing virtually synchronous, stable membership views.
• Efficient and easy to reuse communication patterns for Fireflies ring style gossiping on membership contexts
• Reliable Broadcast — garbage collected, context routed reliable broadcast
• Efficient atomic broadcast in asynchronous networks with byzantine nodes - Consensus
• Dynamic, committee-based, causal ordering service producing linear logs - Replicated State Machines
• JDBC accessible, SQL store backed, materialized views maintained by an SQL state machine. Supports DDL, DML, stored procedures, functions and triggers.
• Google Zanzibar like functionality providing Relation Based Access Control hosted on SQL state machines.

Requirements

Apollo requires JDK 22+ and Maven 3.9.3 and above

Install Maven

The build includes mvnw, and thus there is not need to install Maven. Simply use ./mvnw for all your Maven invocation needs.

See Installing Apache Maven if you prefer to install Maven.

Install GraalVM (Optional)

Apollo optionally requires the GraalVM 22.3.1+ for leveraging Isolates and other fantastic features of the GraalVM. To install the GraalVM, see the Getting Started Guide. For Mac and Apple Silicon, use the Homebrew Tap for GraalVM.

Building Apollo

Important: To provide deterministic SQL execution, Apollo requires an installation step that need only be done once. If you are building Apollo for the first time, you must cd to the root directory of the repository and then:

./mvnw clean install -Ppre -DskipTests

This will perform a full build including the deterministic SQL execution module. After this is complete, you do not need to do this again. You can build Apollo normally without the deterministic SQL module and to do so cd to the root directory of the repository and then:

./mvnw clean install

Note that the install maven goal is required, as this installs the modules in your local repository for use by dependent modules within the rest of the build. You must have invoked maven on the Apollo project root with the " install" goal at least once, to correctly build any arbitrary submodule.

You can use the "--also-make-dependents" argument for maven "-amd" if you want to build a particular module without performing the full build.

Building Apollo Isolate Enclaves

Generation of Apollo shard enclave shared libraries is delegated to the isolate profile:

./mvnw clean install -Pisolates

This will add the isolates modules to the build.

Platform Specific Domain Socket Support

Platform-specific code for supporting Unix Domain Socket in GRPC Netty is segregated into two different modules: domain-epoll and domain-kqueue. These modules are added via platform-specific profiles that are activated for the platform the build is running on.

Modules

Apollo is modularized largely for the purpose of subsystem isolation and reuse. Each module is a Maven module under the source root and contains a README.md documenting (such as it is at the moment, lol) the module.

• CHOAM - Committee maintenance of replicated state machines
• Delphinius - Bare bones Google Zanzibar clone
• Domain-EPoll - linux support for Netty domain sockets
• Domain-KQueue - mac osx support for Netty domain sockets
• Domain-Sockets - unifying abstraction for the different OS domain sockets
• Ethereal - Aleph asynchronous BFT atomic broadcast (consensus block production)
• Fireflies - Byzantine intrusion tolerant, virtually synchronous membership service and secure communications overlay
• Deterministic H2 - Deterministic H2 SQL Database
• Deterministic Liquibase - Deterministic Liquibase
• Gorgoneion - Identity bootstrapping
• Gorgoneion Client - Identity bootstrap client
• Isolates - GraalVM shared library construction of Apollo subdomain enclaves.
• Isolate Functional Testing - Functional testing of Apollo domain enclaves.
• Memberships - Fundamental membership and Context model. Local and MTLS GRPC Routers. Ring communication and gossip patterns.
• Model - Replicated domains. Process and multi-tenant sharding domains and enclaves.
• Protocols - GRPC MTLS service fundamentals, Netflix GRPC and other rate limiters.
• Schemas - Liquibase SQL definitions for other modules
• Sql-State - Replicated SQL state machines running on CHOAM linear logs. JDBC interface.
• Stereotomy - Key Event Receipt Infrastructure. KEL, KERL and other fundamental identity, key and trust management
• Stereotomy Services - GRPC services and protobuf interfaces for KERI services
• Thoth - Decentralized Stereotomy. Distributed hash table storage, protocols and API for managing KERI decentralized identity
• Tron - Compact, sophisticated Finite State Machine model using Java Enums.
• Cryptography - Base cryptography primitives. Bloom filters (of several varieties). Some general utility stuff.

Protobuf and GRPC

Apollo uses Protobuf for all serializations and GRPC for all interprocess communication. This implies code generation. Not something I adore, but not much choice in the matter. GRPC/Proto generation also appears not to play well with the Eclipse IDE Maven integration. To aleviate this, all grpc/proto generation occurs in one module, the aptly named grpc module.

JOOQ

Apollo makes use of JOOQ as an SQL DSL for Java. This also implies code generation and, again, not something I adore. Unlike GRPC, the JOOQ code generation plays very nicely with the Eclipse IDE's Maven integration, so JOOQ code generation is included in the module that defines it.

WIP

Note that Apollo Delphinius is still a work in progress . There is not yet an official release. Thus, it is by no means a full-featured, hardened distributed systems platform.

Requirements

Apollo is a pure Java application The build system is Maven, and requires Maven 3.9.3+. The Maven enforcer plugin enforces dependency convergence, and Apollo is built using Java 22.

Apollo is a multimodule Maven project. This means that the various modules of Apollo are built and versioned as a whole, rather than being separated out into individual repositories. This also means that modules refer to other modules within the project as dependencies, and consequently must be built in the correct order. Note that Maven does this by default, so there should be no issues. However, it does mean that one can't simply cd into a module and build it without building its dependencies first. If you feel you must do so, please make sure to include the "install" goal and please make sure you add the "--also-make-dependents" or " --amd" parameter to your maven invocation.

Code Generation In Apollo

Apollo requires code generation as part of the build. This is performed in the Maven "generate-sources" phase of the build. Consequently, this build phase must be run at least once in order to generate the java sources required by the rest of the build.

The current code generators used in Apollo are GRPC/Proto and JOOQ. GRPC is for the various serializable forms and network protocols used by Apollo. The JOOQ code generation is for the JOOQ SQL functionality.

GRPC/Protoc code generation only occurs in the grpc module and is output into the grpc/target/generated-sources directory. For GRPC/Proto, there are 2 directory roots: grpc/target/generated-sources/protobuf/grpc-java and grpc/target/generated-sources/protobuf/java . For JOOQ, the root directory is (module dir) /target/generated-sources/jooq .

Again, I stress that because these generated source directories are under the "(module dir)/target" directory, they are removed during the "clean" phase of Maven and consequently must be regenerated in order to compile the rest of the build.

Note that adding these generated source directories to the compiler path is automatically taken care of in the Maven pom.xml in the "build-helper" plugin.

IDE Integration

This is Important! Apollo contains one module that create a shaded version of standard libraries. This module must be built ( installed), but only needs to be built once in order to install the resulting jar into your local maven repository. This is performed as part of the top level pom's pre profile. As mentioned previously, this profile must be executed at least once before the full build. Note, however, Eclipse and IntellJ do not understand this transformation and thus will not be able to import this module without errors and messing up the rest of the code that depends on the transformation. What this means is that the IDE thinks the module is fine and doesn't notice there has been package rewriting to avoid conflicts with existing libraries. What this means is that you must exclude this module in your IDE environment. This module will not be imported unless you explicitly do so, so please do not do so. If you really think you need to be working on it, then you probably understand all this. But if you are simply trying to get Apollo into your IDE, importing these modules is gonna ruin your day.

Module to exclude

The module to exclude is:

• h2-deterministic

Again, I stress that you must NOT include this in the import of Apollo into your IDE. You'll be scratching your head and yelling at me about uncompilable code, and I will simply, calmly point you to this part of the readme file.

This module must be built, so please run the following once from the top level of the repository

mvn clean install -Ppre -DskipTests

From the command line before attempting to load the remaining Apollo modules into your IDE. Again, this only needs to be done once as this will be installed in your local Maven repository, and you won't have to do it again. Rebuilding this module will have no adverse effect on the rest of the build.

Eclipse M2E issues with ${os.detected.classifier}

This is a known weirdness with Eclipse M2E with the os-maven-plugin build extension. I've been fine with this, but ran into another project that Eclipse just kept refusing to resolve. I solved this by downloading the supplied maven plugin and adding this to the <ECLIPSE_HOME>/dropins directory. This works because the plugin is also an Eclipse plugin, which is nice.

Your IDE and Maven code generation

Due to the code generation requirements (really, I can't do jack about them, so complaining is silly), the generation phase can occasionally cause interesting issues with your IDE when you import Apollo. I work with Eclipse, and things are relatively fine with the current releases. However, there are sometimes synchronization issues in Eclipse Maven integration that invalidates the generated code and that may require an additional generate-sources pass. Apollo is a multimodule project and be sure you're leaving time for the asynchronous build process to complete.

I have no idea about IntellJ or Visual Code, so you're on your own there.

What I strongly recommend is first building from the command line with -DskipTests - i.e mvn clean install -DskipTests. This will ensure all dependencies are downloaded and all the code generation is complete. Further, if you haven't updated from this repo in a while, don't try to be clever. Delete all the modules from this project from your ide, build/test from the command line and then reimport things. Don't ask for trouble, I always say.

After you do this, you shouldn't have any issue if your IDE Maven integration knows about and takes care of using the build-helper plugin to manage compilation directories for the module in the IDE. However…

Myself, I find that I have to first select the top level Apollo.app module, and then Menu -> Run As -> Maven generate sources (or the equivalent in your IDE). This should generate all the sources required for every submodule, so...

Feel free to generate issues and such, and I will look into it as I do want this to be flawless and a good experience. I know that's impossible, but it undoubtedly can be made better, and PRs are, of course, a thing.

Note that also, for inexplicable reasons, Eclipse Maven will determine it needs to invalidate the grpc generated code and will thus need to be regenerated. I'm trying to figure out the heck is going on, but when this happens, please simply regenerate by selecting the grpc module and performing: Menu -> Run As -> Maven generate sources (or the equivalent in your IDE).

Metrics

Apollo uses Dropwizard Metrics, and these are available for Fireflies, Reliable Broadcast, Ethereal and CHOAM.

Testing

By default, the build uses a reduced number of simulated clients for testing. To enable the larger test suite, use the system property "large_tests". For example

mvn clean install -Dlarge_tests=true

This requires a decent amount of resources, using two orders of magnitude more simulated clients in the tests, with longer serial transaction chains per transactioneer client. This runs fine on my Apple M1max, but this is a beefy machine. YMMV.

Current Status

Currently, the system is in development. Fundamental identity and digest/signature/pubKey encodings have been integrated. Apollo is using Aleph-BFT for consensus, in the form of the Ethereal module. CHOAM has now replaced Consortium, and the SQL replicated state machine now uses CHOAM for its linear log and transaction model.

Multi-tenant shards are in place and being worked upon currently. This integrates Stereotomy and Delphinius using CHOAM. An E2E test of the ReBAC Delphinius service is in development being tested. Full integration of ProcessDomains using Fireflies discovery is in development.