The Problem

There is a disconnect between the state of performance tooling, and the desire teams have to move faster but safer.

On one side, there are several tools to monitor production/staging environments, but on the other side, teams are looking to have full CI/CD integrated where Pull Requests go live upon merge.

This requires that all testing must happen at the Pull Request level on a unique URL per branch and/or commit, and while there are tools that support these types of Performance tests, they all suffer from common problems:

• Single URL.

  • A site is not a single URL. For several sites the most important page is a dynamic route not the homepage.

• Run and block the CI build process.

  • When all tests run on every Pull Request, their intention is to block merging if any issues are found, but during the different stages of development, the single most important task is to get an URL preview. Failing a build on another stage might require a dive into the build dashboard or logs to distinguish errors. Sending a notification, or posting a comment at the end of a specific task can help with this, but it requires additional effort.

• If used for multiple URLs manually, there is no consolidated report.

  • Pull Request pages can be overwhelming at times. From Peer Reviews to comments added by other tools, posting a comment per URL is too noisy.

This Solution

Lightkeeper attempts to solve each one of these issues: 🤞

• Multiple URL support.

  • Configure 1 to many URL routes, from separate domains or extending a base URL.

• Decoupled from the CI build process.

  • It runs when the CI build finishes and is successful. For complex build set ups, the Lightkeeper Bot is available to trigger an event. This allows the build to continue without stalling waiting for a response.

• Consolidated reports, and expandable comments.

  • It posts a consolidated report of all tests, and includes only relevant information in the comment.

Most importantly, Lightkeeper provides granular control of settings per route, from budgets to the Lighthouse endpoint and its options, including chrome flags in the puppeteer configuration.

See Motivation.

Getting Started

Add a lightkeeper.json file in a .github folder. Start from the default configuration.

This is the only file Lightkeeper has access in your code.

There are 3 required fields: baseUrl, ci, and type. Lightkeeper is a budgeting tool, so at least a single type of budget is needed to run sucessfully.

• Replace the baseUrl with a valid URL. Since Pull Request URLs are dynamic, macros are available:

  Macro	Details
  {pr_number}	The Pull Request Number.
  {repo}	A slugified repo name.
  {branch}	A slugified branch name.
  {commit_hash}	The full commit SHA.
  {commit_hash:n}	A trimmed SHA, where n is a digit.
  *{target_url}	The target url from the Github Response.
  **{environment_url}	The url from the Deployment status.

  * Available for statuses and deployments.

  ** Available for deployments only.

• Replace ci and type for your CI tool. Examples:

  Circle CI

  {
    "ci": "circleci",
    "type": "check"
  }
  

  CircleCI uses statuses by default, but also provide a Github Checks integration.

  If multiple checks are running (e.g: Integration Tests, Mobile Tests, etc)

  Use the full name of the specific workflow as it appears in Github, instead of circleci.

  Netlify

  {
    "baseUrl": "{target_url}",
    "ci": "netlify",
    "type": "status"
  }
  

  Travis CI

  {
    "ci": "travis-ci",
    "type": "check"
  }
  

  Zeit Now

  {
    "baseUrl": "{environment_url}",
    "ci": "now",
    "type": "deployment"
  }
  

If you're unsure about the name of your CI tool, it's the name displayed under Developer in the application page: https://github.com/apps/[app-name]

NOTE: These providers and their settings can change. If the provided ci name above does not work, use the full visible name:

Configuration

Visit the wiki for a full list of configuration options.

Additionally, there is a custom configuration example.

Lightkeeper Bot 🤖

Do you have a complex build pipeline that performs several tasks internally? Using the Lightkeeper Bot with the app installed, you can start the process manually without stalling the build:

• Install:

  npm i --save-dev lightkeeperbot

• In your CI tool, run:

  npx lightkeeperbot <baseUrl> [--options]

  Lightkeeper Bot defaults to Travis CI environment variables.

See the full docs.

Tools

• probot
• lighthouse
• chrome-aws-lambda
• puppeteer

Inspiration

• Lighthouse Bot
• Lighthouse GH Reporter

FAQ

• Why isn't this part of the Lighthouse Bot?

  • That was my original intention after reading their FAQ, but after I realized the best format is a Github App along with other features, it was clear the changes would be too drastic. However, Lightkeeper is compatible with their Ligthouse server located at https://builder-dot-lighthouse-ci.appspot.com/ci, if you choose to continue using it.

• Why isn't this a Github Action?

  • I wanted it to be and still do. However, besides the fact that Github Actions are (at the time of writing) Private Beta; it can lead to visual noise when skipping a check.
    
    If in the future, Github could allow a subset level of notifications per event (e.g: check run starts vs check run complete), and possibly a filter on status (only run action when a check run completes and is succesful); I'd be more than happy to switch since it removes the need for a server, and allows for securely sharing secrets.

• I'm getting a Lighthouse error in my reports.

  • If your page fails in Page Speed Insights, it will most likely fail in Lightkeeper too. Consider changing the Lighthouse throttling options, or device emulation.

Related Projects

• Performance Budgets (Docker)
• Zeit Integrations Platform

Contributing

Please open an issue if you have any questions, feature requests or problems, an example configuration for reproduction is deeply appreciatted. 👍

Donating

If you find this tool useful, and want me to spend more time on it. Consider Donating.

AGPL, Copyright (c) 2019 Alfredo Lopez