A web application for managing user experiments for Mozilla Firefox.
Experimenter is a platform for managing experiments in Mozilla Firefox.
Check out the 🌩 Nimbus Documentation Hub or go to the repository that house those docs.
Link | Prod | Staging | Local Dev (Default) |
---|---|---|---|
Legacy Home | experimenter.services.mozilla.com | stage.experimenter.nonprod.dataops.mozgcp.net | https://localhost |
Nimbus Home | /nimbus | /nimbus | /nimbus |
Nimbus REST API | /api/v6/experiments/ | /api/v6/experiments/ | /api/v6/experiments/ |
GQL Playground | /api/v5/nimbus-api-graphql | /api/v5/nimbus-api-graphql | /api/v5/nimbus-api-graphql |
Remote Settings | remote-settings.mozilla.org/v1/admin | remote-settings.allizom.org/v1/admin | http://localhost:8888/v1/admin |
Prerequisites
On all platforms:
On Linux:
On MacOS:
On Windows:
Download from Microsoft store. Or
Download within Powershell.
Open PowerShell as administrator.
Run `wsl --install` to install wsl.
Run `wsl --list --online` to see list of available Ubuntu distributions.
Run `wsl --install -d <distroname>` to install a particular distribution e.g `wsl --install -d Ubuntu-22.04`.
After installation, press Windows Key and search for Ubuntu. Open it and set up username and password.
sudo apt-get update && sudo apt install make
in the ubuntu terminal.
This is necessary for the make secretkey
command and other commands.git --version
. If it's not recognized, install git using sudo apt install git
Clone the repo
git clone <your fork>
Copy the sample env file
cp .env.sample .env
Set DEBUG=True for local development
vi .env
Create a new secret key and put it in .env
make secretkey
vi .env
...
SECRETKEY=mynewsecretkey
...
Run tests
make check
Setup the database
make refresh
Run a dev instance
make up
Navigate to it and add an SSL exception to your browser
https://localhost/
One might choose the semi dockerized approach for:
Notes:
Pre reqs macOS instructions:
brew install postgresql llvm openssl yarn
echo 'export PATH="/usr/local/opt/llvm/bin:$PATH"' >> ~/.bash_profile
export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/opt/openssl/lib/
Ubuntu 20.04 instructions:
# general deps (also see `poetry` link above)
sudo apt install postgresql llvm openssl yarn
# add'l deps* for poetry / python setup
sudo apt install libpq5=12.9-0ubuntu0.20.04.1
sudo apt install libpq-dev
*Notes
poetry install
(next step) requires python 3.9, but there are multiple options for resolving this, see here
Install dependencies
source .env
cd experimenter
poetry install # see note above
yarn install
env values
.env (set at root):
DEBUG=True
DB_HOST=localhost
HOSTNAME=localhost
Start postgresql, redis, autograph, kinto
make up_db (from project root)
Django app
# in experimenter
poetry shell
yarn workspace @experimenter/nimbus-ui build
yarn workspace @experimenter/core build
# run in separate shells (`poetry shell` in each)
yarn workspace @experimenter/nimbus-ui start
./manage.py runserver 0.0.0.0:7001
Pro-tip: we have had at least one large code refactor. You can ignore specific large commits when blaming by setting the Git config's ignoreRevsFile
to .git-blame-ignore-revs
:
git config blame.ignoreRevsFile .git-blame-ignore-revs
If using VSCode, configure workspace folders
Add /experimenter/
and /experimenter/experimenter
folders to your workspace (File -> Add Folder to Workspace -> path/to/experimenter/experimenter
)
From the /experimenter/experimenter
folder, run yarn install
Make sure you are using the correct version of node
node -v
Troubleshooting:
npm cache clean --force
On certain pages an API endpoint is called to receive experiment analysis data from Jetstream to display visualization tables. To see experiment visualization data, you must provide GCP credentials.
Prequisites
moz-fx-data-experiments
#nimbus-dev
for a project admin to grant storage.objects.list
permissions on the moz-fx-data-experiments
projectAuthorize CLI with your account
make auth_gcloud
Error saving Application Default Credentials: Unable to write file [...]: [Errno 21] Is a directory: ...
, delete the directory and try again (rm -rf ~/.config/gcloud
)The next time you rebuild the docker-compose environment, your credentials will be loaded as a volume
make refresh
)(optional) Verify access
make refresh
make bash
./manage.py shell
from django.core.files.storage import default_storage
default_storage.listdir('/')
We support user uploads of media (e.g. screenshots) for some features.
In local development, the default is to store these files in /experimenter/media
using Django's FileSystemStorage
class and the MEDIA_ROOT
and MEDIA_URL
settings.
In production, a GCP bucket and credentials are required.
The bucket name is configured with the UPLOADS_GS_BUCKET_NAME
setting. For example:
UPLOADS_GS_BUCKET_NAME=nimbus-experimenter-media-dev-uploads
For local testing of a production-like environment, The credentials should be configured as described in the previous section on Google Credentials for Jetstream.
In the real production deployment, credentials are configured via workload identity in Google Kubernetes Engine.
Experimenter uses docker for all development, testing, and deployment.
Build the application container by executing the build script
Build the supporting services (nginx, postgresql) defined in the compose file
Create dummy SSL certs to use the dev server over a locally secure connection. This helps test client behaviour with a secure connection. This task is run automatically when needed.
Stop and delete all docker containers. WARNING: this will remove your database and all data. Use this to reset your dev environment.
Apply all django migrations to the database. This must be run after removing database volumes before starting a dev instance.
Populates the database with dummy experiments of all types/statuses using the test factories
Run kill, migrate, load_locales_countries load_dummy_experiments. Useful for resetting your dev environment when switching branches or after package updates.
Cirrus is required to run and test web application experiments locally. It is disabled by default. To enable Cirrus run:
export CIRRUS=1
This will be done automatically for any Cirrus related make commands.
Start a dev server listening on port 80 using the Django runserver. It is useful to run make refresh
first to ensure your database is up to date with the latest migrations and test data.
Start postgresql, redis, autograph, kinto on their respective ports to allow running the Django runserver and yarn watchers locally (non containerized)
Start Django runserver, Celery worker, postgresql, redis, autograph, kinto on their respective ports to allow running the yarn watchers locally (non containerized)
Start all containers in the background (not attached to shell). They can be stopped using make kill
.
Pull in the latest Kinto Docker image. Kinto is not automatically updated when new versions are available, so this command can be used occasionally to stay in sync.
Run all test and lint suites, this is run in CI on all PRs and deploys.
If you have a test failing to find an element (or finding too many, etc.) and the DOM is being cut off in the console output,
you can increase how much is printed by locally editing the DEBUG_PRINT_LIMIT=7000
in the Makefile
(line starts with JS_TEST_NIMBUS_UI
).
Run only the python test suite.
Start a bash shell inside the container. This lets you interact with the containerized filesystem and run Django management commands.
You can run the entire python test suite without coverage using the Django test runner:
./manage.py test
For faster performance you can run all tests in parallel:
./manage.py test --parallel
You can run only the tests in a certain module by specifying its Python import path:
./manage.py test experimenter.experiments.tests.api.v5.test_serializers
For more details on running Django tests refer to the Django test documentation
To debug a test, you can use ipdb by placing this snippet anywhere in your code, such as within a test method or inside some application logic:
import ipdb
ipdb.set_trace()
Then invoke the test using its full path:
./manage.py test experimenter.some_module.tests.some_test_file.SomeTestClass.test_some_thing
And you will enter an interactive iPython shell at the point where you placed the ipdb snippet, allowing you to introspect variables and call methods
For coverage you can use pytest, which will run all the python tests and track their coverage, but it is slower than using the Django test runner:
pytest --cov --cov-report term-missing
You can also enter a Python shell to import and interact with code directly, for example:
./manage.py shell
And then you can import and execute arbitrary code:
from experimenter.experiments.models import NimbusExperiment
from experimenter.experiments.tests.factories import NimbusExperimentFactory
from experimenter.kinto.tasks import nimbus_push_experiment_to_kinto
experiment = NimbusExperimentFactory.create_with_status(NimbusExperiment.Status.DRAFT, name="Look at me, I'm Mr Experiment")
nimbus_push_experiment_to_kinto(experiment.id)
You can also interact with the yarn commands, such as checking TypeScript for Nimbus UI:
yarn workspace @experimenter/nimbus-ui lint:tsc
Or the test suite for Nimbus UI:
yarn workspace @experimenter/nimbus-ui test:cov
For a full reference of all the common commands that can be run inside the container, refer to this section of the Makefile
Run the integration test suite for experimenter inside a containerized instance of Firefox. You must also be already running a make up
dev instance in another shell to run the integration tests.
Run the integration test suite for nimbus inside a containerized instance of Firefox. You must also be already running a make up
dev instance in another shell to run the integration tests.
FIREFOX_VERSION should either be nimbus-firefox-release
or nimbus-firefox-beta
. If you want to run your tests against nightly, please set the variable UPDATE_FIREFOX_VERSION
to true
and include it in the make command.
Run the Nimbus SDK integration tests, which tests the advanced targeting configurations against the Nimbus SDK.
FIREFOX_VERSION should either be nimbus-firefox-release
or nimbus-firefox-beta
. If you want to run your tests against nightly, please set the variable UPDATE_FIREFOX_VERSION
to true
and include it in the make command.
First start a prod instance of Experimenter with:
make refresh&&make up_prod_detached
Then start the VNC service:
make FIREFOX_VERSION integration_vnc_up
Then open your VNC client (Safari does this on OSX or just use VNC Viewer) and open vnc://localhost:5900
with password secret
. Right click on the desktop and select Applications > Shell > Bash
and enter:
cd experimenter
sudo apt get update
sudo apt install tox
chmod a+rwx tests/integration/.tox
tox -c tests/integration/ -e integration-test-nimbus
This should run the integration tests and watch them run in a Firefox instance you can watch and interact with.
To use NoVNC, navgate to this url http://localhost:7902
with the password secret
. Then you can follow the same steps as above.
An example using PYTEST_ARGS to run one test.
make integration_test_legacy PYTEST_ARGS="-k test_addon_rollout_experiment_e2e"
Note: You need the following firefox version flag when running integration tests
FIREFOX_VERSION=nimbus-firefox-release
An example for above:
make FIREFOX_VERSION=nimbus-firefox-release integration_test_nimbus PYTEST_ARGS=ktest_rollout_create_and_update
This builds and sets up the mobile sdk for use in testing.
Navigate to experimenter/tests/tools
To test a targeting expression, first add an app context named app_context.json
to the experimenter/tests/tools
directory.
You can then invoke the script with the --targeting-string
flag:
python sdk_eval_check.py --targeting-string "(app_version|versionCompare('106.*') <= 0) && (is_already_enrolled)"
The script should return the results, either True
, False
, or an error.
Note that you can change the app_context
live, and run the script again after.
In development you may wish to approve or reject changes to experiments as if they were on Remote Settings. You can do so here: http://localhost:8888/v1/admin/
There are three accounts you can log into Kinto with depending on what you want to do:
admin
/ admin
- This account has permission to view and edit all of the collections.experimenter
/ experimenter
- This account is used by Experimenter to push its changes to Remote Settings and mark them for review.review
/ review
- This account should generally be used by developers testing the workflow, it can be used to approve/reject changes pushed from Experimenter.The admin
and review
credentials are hard-coded here, and the experimenter
credentials can be found or updated in your .env
file under KINTO_USER
and KINTO_PASS
.
Any change in remote settings requires two accounts:
Any of the accounts above can be used for any of those two roles, but your local Experimenter will be configured to make its changes through the experimenter
account, so that account can't also be used to approve/reject those changes, hence the existence of the review
account.
For more detailed information on the Remote Settings integration please see the Kinto module documentation.
Experimenter has two front-end UIs:
core
is the legacy UI used for Experimenter intake which will remain until nimbus-ui
supersedes itnimbus-ui
is the Nimbus Console UI for Experimenter that is actively being developedLearn more about the organization of these UIs here.
Also see the nimbus-ui README for relevent Nimbus documentation.
API documentation can be found here
Please see our Contributing Guidelines
Experimenter uses the Mozilla Public License