Kenobi Save

Kenobi -- Open Source Log-to-Metrics & Log-to-Funnels platform

Kenobi helps you proactively monitor your application by defining correlations in your logs. Kenobi is designed to have a flexible rule-engine, easy-to-use interface and hassle-free deployment.

Capabilities

1. Log to events:

You can buffer a stream of logs to Kenobi and define filters and transformations on the platform using the GROK parser built into the platform.

2. Event Sequences:

Once you have events flowing in, you can define different sequences by joining them. Here are some of the examples:

1. 2-step sequence (defined as a monitor): Define a relation, A₁ → A₂ with A₁ being primary and A₂ being secondary event
2. n-step sequence (defined as a funnel): Define a set of events A₁ → A₂ → ... → A_n that are expected to happen sequentially
   • todo -- add recording here.
3. n-step trees (defined as an entity): Define a set of events that are not necessarily sequential but inter-related
   • todo -- add recording here

3. Metrics:

Define aggregations on your events and sequences. Some of the allowed metric types include:

• Transition time and success rate between nodes in an event sequence or tree
• Aggregation functions on any attribute within an event
• Aggregation functions on any attribute within an event, grouped by another (enum) attribute in any other event

4. Alerting Rule Engine:

Kenobi enables you to create rules over your events and event sequences. Here are some rules that would be possible in the platform:

• Send me an alert if node A₂ does not happen after node A₁ for every instance where attribute_value=specific_value.
• Send me an alert if more than 5% of the nodes "between" node A₁ and node A_n are stuck at a specific node.
• Send me an alert if sequence is stuck at node A_i for more than stipulated duration.

Kenobi can push these alerts to itself, email and slack currently.

Sounds useful?

Play around in demo environment 👇🏽

Exploring Kenobi in a Sandbox

For the purpose of this sandbox, we have created a sample payment application. In the sandbox, you will be able to see logs from one such application, and how we can transform it into funnels and charts.

• Cloud Sandbox: Play around in the cloud sandbox here (email ID required - we will not share your PII with any external party or send you unnecessary emails)
• Self-hosted Sandbox: To deploy Kenobi in your environment, run the following on a Linux machine with Docker (recommended 8GB memory):

 /bin/bash docker_deploy.sh

After this, you can use the platform on port 80 on your host IP address. Use the following credentials for login:

Username -> [email protected]
Password -> password

Take note of the deployment hostname/IP and note the API token from the API Keys section in your portal. They will be handy when setting up integration for sending events.

Doctor Droid's open source deployment is reasonably scalable due to its micro-service architecture. In case you see difficulties in scaling it, reach out to us at [email protected] and we'll help you.

Creating sample events

You can run the in-built script for a basic payments workflow in an ecommerce company. Once your deployment is setup, set the following environment variables:

export HOST_URL=<your_deployment_ip_or_hostname> # Add http or https in it without the trailing slash
export API_TOKEN=<your_api_key>

Post this, you can run the basic python script that publishes events. Make sure you have python3 and pip3 installed in your environment

/bin/bash events_publish.sh

After your events start coming in, you can check them in the Search section.

Integrating your log stream to Kenobi

Before being able to create funnels, metrics or rules, you need to parse data into a kenobi readable format.

You can stream application logs into Kenobi using:

• Native events SDK (this SDK is compatible to the events format expected on the platform) In case you want to stream from your existing sources, you can read the following documentations:
• Cloudwatch Logs via Firehose
• Segment events via Firehose

How to stop the deployment

Run the following command from the root directory

 /bin/bash docker_stop.sh

Cloud Hosting

Doctor Droid supports a robust cloud platform for Kenobi. If you'd like to use the cloud platform instead of managing the platform in-house, sign up on our website or book a demo.

License

This repo is available under the MIT license.