Emcee is a tool that runs Android and iOS tests in parallel using multiple simulators and emulators across many servers
timeWeighted
test splitter that works hand in hand with test history service and provides the shortest possible time for entire test run. See more in doc (iOS, Android)--trace
option. See more in emceeFree runTests --help
equally divided
retry on worker
silentWorkerAutomaticRestartFeatureConfig
Common:
iOS:
runtimeXCTest
. Use it for discovering dynamically added tests with frameworks like Quick, Kiwi, etcscheduleTests
command dispatches tests to previously deployed queue. It's useful in systems where Emcee gets deployed with external platform operatorAndroid:
videoRecordStrategy
Cloud:
Minor fixes:
retryOnWorker
tests retry mode does not cause excessive retries on queueSince version 20.0.0 Emcee we will change a minor version for ongoing regular releases. Major versions will be increased with significant changes or broad refactoring.
We are excited to announce that Emcee now supports the Android platform. Run Emcee queue and workers from container images in Docker or your Kubernetes cluster to shorten testing time. Keep calm, we continue to develop iOS part as well.
Android workers, as well as their iOS counterparts, receive test jobs from queue. It means you get smart bucket slicing strategies, test retries, and better hardware utilization out of the box.
xcodebuild build-for-testing -testPlan <name>
and then feed the resulting xctestrun file to Emcee via --xctestrun
. See more in Emcee runTests --help
.18th release is concentrated on fixing bugs, speeding up things, and separating the functionality between features which become available after purchasing a licence and always-free version.
Starting from version 17, Emcee is distributed as-is as a closed source project and available in a form of prebuilt binaries. Version 18 goes beyond this and introduces two distinct products: free Emcee and paid Emcee.
Free version (available as a download on Github) allows you to test the product with up to 3 workers, however, you'll need to buy licenses for some richer functionality like:
Main functionality remains available in free version:
The current source code has been frozen from April 2022, which was version 16. We will keep it available as is, and we're not planning on removing it. All made forks will still be present obviously as well, as we can't control them.
Version 17 is closed source, yet fully functional. You can use it without any limitations as well.
We haven't developed any plans on upstreaming changes from v17 into open source yet, however, we are considering it.
This is huge one and helpful. Emcee now tracks if job is needed at all, and automatically deletes it if it is not needed to run anymore. This helps on CI: when you cancel the test run, job gets deleted automatically.
Tech note: previously Emcee client process issued a /jobDelete
request which sometimes could get lost; now queue itself tracks all jobs.
Emcee now attempts to restart its workers in case if they die. So if you decided to reboot your machine, or if it kernel panics, it will become a worker automatically.
Emcee worker can now clone simulators instead of booting a new one. This speeds up creation quite a bit, and saves disk space. Emcee will clone any previously shut down simulator it booted. If Emcee never booted suitable simulator, it will create one instead. We advise to use smaller values of automaticSimulatorShutdown
and automaticSimulatorDelete
settings, like 120 seconds, to have better effect of simulator cloning.
As an example, booting a new simulator can take 1-2 minutes easily, but cloning and booting takes around 40 seconds.
You can use the new, a little simplified plugin SDK which is available here: https://github.com/avito-tech/EmceePluginSupport
It's easier to wire up everything, as there are only two exposed targets now: EmceePlugin and EmceePluginModels.
Boilerplate code has been reduced just to:
let plugin = try Plugin { (event: PluginAppleTestEvent) in
// process an event
}
plugin.join()
/globalQueueState
. Maybe this will be helpful for you to debug how queue is being formed when you submit your jobs.emcee.workerid
and the value is actually worker ID.--device
and --runtime
arguments now accept only fully qualified Core Simulator IDs, e.g. com.apple.CoreSimulator.SimRuntime.iOS-16-4
and com.apple.CoreSimulator.SimDeviceType.iPhone-14
. Please refer to xcrun simctl list
for more examples.runTests
command, set them in the calling environment with a EMCEE_
prefix.emcee.cpu.architecture
, the value is CPU arch returned by uname -a
, e.g. arm64
.testFinished
event is now correctly delivered to plugins when tests times out.workerPortRange
which controls exactly that. You can also set queue port range via queuePortRange
.discoverAppleTests
lets you run test discovery much easier than before, without composing a test arg fileA decent place to discuss knowledge about Emcee and obtain a licence!
Russian: https://t.me/emcee_ios
English: https://t.me/emcee_ios_en
This release brings long awaited feature - generating a resulting xcresult
file which you can open in Xcode right after running your tests. Just use runTests
command and provide --result-bundle <path>
to generate your result bundle.
You can also control the lifetime of xcresult
attachments via TestArgFile.entries.testAttachmentLifetime
- it can be deleteOnSuccess
, keepAlways
, keepNever
. Just like xcodebuild
, huh!
application/json: Content-Type
which swift-nio thing forcefully rejects; now it is set correctly to Content-Type: application/json
.Less memory leaks and more speed thanks to updating HTTP server to Vapor
Less logs are generated! Again!
Emcee now kills xcodebuild
with more suitable signal SIGINT
which leads to less result bundle corruptions.
libssh
freedom has arrived! Emcee does not require it to be installed on your system. Also, it is adviced to use key-based authentication.
We got rid of duplicate errors from junit. We got rid of duplicate errors from junit.
Help (e.g. Emcee runTests -h
) got even better thanks to more information provided.
You can now configure simulator count on each worker, locale, keyboard, language via runTests
command. Please refer to Emcee runTests -h
.
A decent place to discuss and share your knowledge about Emcee!
Russian: https://t.me/emcee_ios
English: https://t.me/emcee_ios_en
We've conducted a bunch of interviews with teams who use Emcee. We were provided with nice feedback which we used to improve our product. In this release, we focus on ease of use and an easy first try experience.
It is available right at README.
We introduce a new command - runTests
. A very similar one to a famous runTestsOnRemoteQueue
but it is dedicated to opening the world of simplicity and clear setup for most Emcee users. It has the following syntax and options:
$ Emcee runTests -h
--queue
: describes where should a queue be started. This argument may be repeated multiple times, in which case Emcee will use hosts one by one until it manages to start the queue. You must provide at least one value here.--worker
: describes where worker should be started. This argument may be repeated multiple times. It is required to provide at least one value.--device
: Device to run test on, e.g. iPhone X
. Required. Note: iPhone SE
generations might be tricky to use!--runtime
: Runtime to run test on, e.g. 15.0
. Required.--test-bundle
: Location of an .xctest
bundle which tests will be executed. Required.--app
: Location of an .app
bundle. Optional. Please refer to our guide to learn more about tests with host app.--runner
: Location of an XX-Runner.app
. Optional.--test
: Test to execute, e.g. ClassName/testMethod
. Optional. If not specified, Emcee will run all tests. You can specify multiple tests to execute, e.g. --test Class/test1 --test Class/test2
--retries
: How many retries to attempt to run each test successfully. Optional. By default it will retry tests once if failure happens.--test-timeout
: Maximum test execution duration. Optional. Default test duration limit is 180 seconds.--junit
: Path where the Junit report file should be created. Optional.--trace
: Path where the Chrome trace file should be created. Optional.Let's explain --queue
and --worker
arguments. These are URLs which describe how to connect to a machine. Currently only SSH is supported. Some examples:
--queue ssh://emcee:[email protected]/Users/emcee/emceequeue.noindex
: it means to run Emcee queue on a queue.example.com
host, by authenticating with emcee
username and pass
password. The working directory will be located at /Users/emcee/emceequeue.noindex
.--worker ssh://[email protected]/Users/emcee/emceeworker.noindex?custom_rsa
: it means the queue will start its worker on worker.example.com
. Worker will use /Users/emcee/emceeworker.noindex
as its working directory. Also, Emcee will use emcee
as username and ~/.ssh/custom_rsa
key to auth against a worker host. This key is expected to be present on queue host.--worker ssh://[email protected]/Users/emcee/emceeworker.noindex#/absolute/path/to/custom_rsa
: same as above, but queue will use a key from /absolute/path/to/custom_rsa
.Emcee now contains a built-in HTTP server that will cover most of your needs. This significantly simplifies the flow and lowers the entry threshold.
You can pass local paths into runTests
command. Provided local .app
and .xctest
bundles will be hosted via a built-in HTTP server transparently, allowing all workers to download artifacts in order to run tests.
You can also provide local paths inside test arg file which you pass into runTestsOnRemoteQueue
, those will also be hosted via the built-in server. Handy!
And finally, you can provide a local queue server configuration file into runTestsOnRemoteQueue
. Now there is no need to upload it to a HTTP server.
-h
π‘You can now get help for any command by typing -h
next to it, e.g. Emcee -h
or Emcee runTests -h
. Sounds like Emcee is ready for such kind of hi-tech.
Should you provide an incorrect test arg file or queue configuration JSON file, Emcee will now print human-readable errors. You will be amazed. Here is one error for your consideration:
Failed to decode value for key "entries" in "test arg file": key "xcTestBundle" not found at entries[0].buildArtifacts
That is a crystal clear explanation of what went wrong, is it not?
jobId
βοΈIt wasn't clear for some of our users that jobId
field in a test arg file should be unique. Now you can omit it, Emcee will generate a random jobId
for you.
You can now specify how you'd like your tests to be retried if they fail. In your test arg file's entries
there is a new field testRetryMode
. Possible values:
retryThroughQueue
β if a test fails on a worker, it will be returned back to the queue. A queue will then retry this test on other workers, up to the specified number of retries. This is the default behavior. This allows the test to be executed on different hosts, potentially eliminating the problem of a broken environment (because every macOS installation in Universe is unique).
retryOnWorker
β if the test fails, the same worker will perform retry.
There was a bug (OMG) that resulted in redundant simulator patching (read: apply simulator settings). Now Emcee won't patch simulators if everything is patched.
There is a way to disable worker cache. In queue server configuration, in worker specific settings, next to numberOfSimulators
field there are two more fields:
maximumCacheSize
β maximum cache size in bytes.maximumCacheTTL
β maximum TTL for any cached item, in seconds.You can set 0
these fields, and it will disable cache entirely. This is handy if you want to debug something.
You can now stop workers from removing all test artifacts automatically after a test finishes and plugins terminate. In a test arg file's entries
there is a new field runnerWasteCleanupPolicy
. Possible values:
clean
β default value, it means everything will be deletedkeep
β all created files will be preserved.You can now control what logs should be captured during a test run. These logs are obtained from xcresult
bundles after the tests finish. In the test arg file's entries
there is a new field logCapturingMode
. Possible values:
allLogs
β all log messages will be capturedonlyCrashLogs
β only logs that look like crash logs will be capturesnoLogs
β no logs will be captureddyld
You can now insert dynamic libraries into a test. In a test arg file's entries
there is a new field userInsertedLibraries
. This is an array of paths. These paths will be passed as DYLD_INSERT_LIBRARIES
env into the test. Combine it with state of the art macOS imaging features, with a fact that tests are being executed via xcodebuild
(so it can expand __TESTBUNDLE__
and other strings - look at man xcodebuild.xctestrun
), and this will provide you some additional flexibility for your test execution process.
libssh
, so you can use it with ease.Emcee
binary from GitHub, do not forget to remove quarantine attribute: xattr -c Emcee
Full Changelog: https://github.com/avito-tech/Emcee/compare/15.0.0...16.0.0
Today we celebrate a bit delayed Emcee v15 release π₯³
Emcee queues can be run on multiple hosts now, allowing even more graceful degradation when necessary. For example, if you set your queue host on maintenance (turn it off, or if aliens destroy your machine), queue may now be started on a spare host(s), allowing you to keep running your tests. To use this, just provide multiple queue hosts in queue_config.json
via queueServerDeploymentDestinations
array. Emcee iterate over the hosts in provided order.
xcodebuild
now deletes all attachments π from resulting xcresult
bundle if test was successful β
, reducing a load on the workers. Thanks again to @EvgenyIv96 for implementing this.didShowContinuousPathIntroduction
to true
in your test arg file.xcresult
files and extracts test failures from there in addition to parsing result stream. This improves test error capturing significantly. π‘Emcee initTestArgFile
command creates a sample test arg file which you can fill with your settings. Useful for quick-start! π₯©Emcee initQueueServerConfig
command creates a sample queue config. Again, useful for initial setup. π±sudo
. You can create a standard user with limited rights and make Emcee use it. π For this scenario, Emcee now checks if ~/L/D/CoreSimulator
folder exists, and creates it if it is missing.Just don't forget to install libssh2
via excellent package manager which uses chemistry, water supply, liquid manipulation, and drink-related domain terms to describe its all possible actions and processes. β’οΈ
Also, do not forget about clearly understandable fact that Apple Silicon chips have special destination path for installable things in that honorable package manager, and that it is different on Intel chips. Thus, it is unlikely you will be able to use Intel binary on Apple Silicon Mac, which is very due to extreme quality of any modern software. π Please render this fine emoji in red in your mind.
Full Changelog: https://github.com/avito-tech/Emcee/compare/14.0.0...15.0.0
xcresult
-eys into Plugins@venigreat implemented a feature that allows to access xcresult
bundles from plugins, allowing to process them on the go. TestContext.environment
now contains a new entry which points to a xcresult
bundle. You can and should use XcodebuildTestRunnerConstants.envXcresultPath
to get the name of that env.
By the way, there is an open source Allure plugin for Emcee
@venigreat implemented the following feature: Emcee now allows to access SSH by using private key files. Previously it only used password based auth. You specify auth mechanism via queue configuration.
@venigreat implemented a third feature: Emcee now can add arbitrary HTTP headers to its network requests. This is useful if your storage (e.g. artifactory, nexus, etc.) requires authentication via headers, or if your back-end wants to track its clients via some X-Fancy-Header
value.
In this release, we focused on adding support for distributed logging system, and we used Kibana for these purposes. Thus, we decided to document how Emcee logging system works, and provide in-depth explanation of what metrics Emcee reports.
Emcee now respects two analytics configurations. One is a global one. It gets captured by the queue when it starts (queueServerConfiguration.globalAnalyticsConfiguration
), and it is used to report some global events which do not relate to any specific job. Some examples are:
Another analytics configuration is a specific one. You provide a configuration for it via test arg file, and it is used to report job-specific analytic events. Some, if not all, examples:
Job preparation duration - how long to takes to prepare job, i.e. discover all tests and enqueue them
Bucket processing duration. Each job consists of tests, and tests are grouped into buckets. This metric is used to report the durations of these buckets.
Stuck buckets in job
Job queue states - number of enqueued and dequeued buckets, time to dequeue a bucket
Test discovery events - test case count, test count, and aggregated test discovery duration
Time to start a test metric
Test preflight duration
Test postflight duration
It is worth noting that specific analytics configuration does NOT inherit anything from global configuration. You must pass complete configurations via both queue configuration and via test arg file. One thing to keep in mind is that test arg file should contain a job specific analytics configuration, so at least analyticsConfiguration.persistentMetricsJobId
can (and probably should) be different depending on your job (e.g. unit tests, ui tests, full regression suite, etc.).
As usual, it is better to refer to the unit tests to see all possible ways how test arg file can be configured. Tests for root test arg file structure are here, and tests for test arg file entries are here.
Emcee now supports sending all its logs to a common Kibana end point(s). Kibana is configured via [global]AnalyticsConfiguration.kibanaConfiguration
. Emcee will add the following values to logged messages:
hostname
β host which emitted a log messageemceeVersion
β version of EmceeemceeCommand
β command being executed, e.g. dump
, runTestsOnRemoteQueue
, distWork
, etc.workerId
β worker which emitted a log message.processId
and processName
β pid and process name of Emcee itself, e.g. 3224
and EmceeWorker_ab33da
.subprocessId
and subprocessName
β pid and process name of a subprocess which Emcee executedxcrunToolName
β tool name which has been executed via xcrun
, e.g. for xcrun xcodebuild
this key will have xcodebuild
valuepersistentMetricsJobId
β a job id from test arg filesubprocessPipe
β standard output pipe name for messages logged by subprocesses, e.g. stdout
or stderr
. Currently Emcee logs subprocess messages during test discovery process, but more cases may be adopted in the future.In addition to default keys that Emcee adds automatically, you may add your keys via [global]analyticsConfiguration.metadata
. On our CI, for instance, we add the following keys to help us track down job-specific logs (so we append these values to specific configuration via testargfile): ciBranch
, ciBuildNumber
, ciPullRequestId
.
--temp-folder
is now optionalTestsWorkingDirectorySupport
), but as soon as all plugins terminate, Emcee worker will clean up all files. This fixes an issue with a leaking disk space.xcodebuild -resultStreamPath
file contents by default now and even attempts to add crash logs (if xcodebuild
will emit them).xcrun simctl spawn
calls are now limited by a hardcoded timeout of 30 seconds to prevent potential stalls. 30 seconds is more than enough, but if it is not, reach out to us.testArgFile.testTimeoutConfiguration.testRunnerMaximumSilenceDuration
(https://github.com/avito-tech/Emcee/wiki/Test-Arg-File#testtimeoutconfiguration).xcodebuild
test runner. Support for fbxctest
has been removed entirely. Also, we don't parse xcodebuild
logs anymore, we always use the secret power of -resultStreamPath
flag.