Emcee is a tool that runs Android and iOS tests in parallel using multiple simulators and emulators across many servers
This is a secret technology made by Apple allows you to get real time updates on your test execution process. All you need to do is to pass -resultStreamPath <exiting_file> to xcodebuild and it will start dumping JSON stream right into that file.
We've reverse-engineered models from that file. The results of this job are the ResultStream and ResultStreamModels modules if you want to use it in your projects.
If you want to use it in Emcee though, we've covered it under a FEATURE TOGGLE — that's right, you'll have to opt-in (for now):
EMCEE_USE_RESULT_STREAM = true to your test enviornments (via test arg file) and your tests will be executed via this new high technology.Emcee kickstart command will now correctly start a worker even if one never started before
JSONStream CPU performance improved by 3.5x (or 350%). Also, we now use bytes (UInt8) instead of unicode scalars. Only UTF8 encoding is supported though.
TestsWorkingDirectorySupport target. Use it by depending on EmceeInterfaces
New metric! Useless xcodebuild invocations. This event describes a situation when Emcee invokes xcodebuild to run the tests, but it never runs them, instead waits for something (e.g. testmanagerd) and then terminates. Usually this happens when too much simulators are being in use on a machine. Reducing simulator count or rebooting your machine may help with reducing useless invocations.
Fixed dequeud/enqueued metric values
New metric reporting system! Emcee can now report to Statsd! Analytics configuration accepts new settings for it.
When you enqueue tests with non-satisfiable worker requirements, these tests will be marked as failed if no worker is available to run these tests instead of allowing queue to keep them enqueued infinitely while waiting for suitable worker to appear. 🦺
Fixed running Emcee on larger machine farms with more than 64 computers. 🤡
Scheduling tests with empty worker requirements now again supported. 🦵🏻
[Synchronous]QueueClient classes have finally been deleted.
JobStateFetcher for fetching current job state, JobResultsFetcher for fetching job results, JobDeleter for deleting job are the new APIs of QueueClient module.
By default. Read more about this feature here.
New feature! Allows to schedule tests to workers that do meet specific requirements. Read more about this feature here. Currently only available Xcode versions are exposed, but this can be extended. Submit your ideas.
New feature! Allows you to send a kickstart command to the queue via REST to make it attempt revive the specific workers. Useful if you kill EmceeWorker process for machine maintenance and then need to get it back to the working state.
All metrics now have emcee version in them
{
"jobId": "jobId",
"entries": [
{
"testsToRun": ["all"],
"testDestination": {"deviceType": "iPhone X", "runtime": "11.3"},
"testType": "uiTest",
"buildArtifacts": {
"appBundle": "http://example.com/App.zip#MyApp/MyApp.app",
"runner": "http://example.com/App.zip#Tests/UITests-Runner.app",
"xcTestBundle": "http://example.com/App.zip#Tests/UITests-Runner.app/PlugIns/UITests.xctest"
}
}
]
}
Made --emcee-version non optional - this is generated automatically when you do make build.
Removed --queue-server-destination command line arg from runTestsOnRemoteQueue command. This file has been merged into --queue-server-configuration. By the way, --queue-server-run-configuration became --queue-server-configuration (no stupid run anymore).
--job-id, --job-group-id, --priority, --job-group-priority arguments has been removed. They all migrated to a test arg file.
Models target has been removed from Swift package.
Demangling buffer size (this is related to test discovery) has been increased from 1K to 10K. Should be "enough for everyone". If it fires, let us know, we can implement dynamic buffer size.
EmceeAdmin allows you to observe and control your Emcee workers via GUI. The latest release supports matching Emcee workers and TeamCity agents.
EmceeCommunications library, a set of APIs to discover and talk to Emcee queue.enableWorker command correctly enables worker now
Emcee cache size is now limited to 20Gb by default. In your plugins, if you use cache, set this value to 20Gb as well.
Log files are cleaned every 3 hours instead of cleaning them on each Emcee invocation.
Improve log contents for better readability and usefulness
Fix partial data parsing from test runner
This feature has been implemented for fbxctest previously, and didn't work for xcodebuild test runner. Now this has changed! Emcee correctly patches simulator plist files, and restart affected daemons to ensure patched settings are applied. There is even a new keyboard setting for you to patch - didShowContinuousPathIntroduction, which allows to skip annoying gesture-based input hint.
Emcee now gets simulator for its actual state instead of keeping it in memory. This allows to avoid some nasty bugs, e.g. when Emcee attempts to shutdown simulator, but the operation takes too long, so Emcee kills xcrun simctl shutdown process in order to proceed, but actually simulator is eventully getting torn down by the Apple mechanics. This led to inconsistent state; not anymore.
test.between_tests.duration - provides a duration between "test has finished" and "test started" events in the same bucket.simulator.allocation.duration - provides a duration of how long simulator gets prepared to be used for testingtest.preflight - provides a duration of how long test runner (e.g. xcodebuild) takes to prepare itself and start running a first test. More precisely, this is a time from when test runner process starts till the time when first test starts.test.postflight - this is a duration between test finish event and the termination of test runner process. Describes situation when test runner process finishes running all tests, but does not terminate, wrapping up its work.~/Library/Logs/ru.avito.emcee.logs/ like logs of Emcee itself. This is useful for investigation. Emcee cleans all logs that are older than 30 days when it launches, but you may clean up these logs yourself if you don't need them.runTestsOnRemoteQueue, Emcee now prints number of enqueued and dequeued tests instead of buckets, as well as what tests each worker is executing. 🔥xcodebuild as a test runner.In our environment Emcee was able to perform our complete set of about ~2k unit, ~1.2k application, and ~600 UI tests using pure xcodebuild test runner. This means you can stop providing fbxctest to Emcee and use "testRunnerTool": {"type": "xcodebuild"} in test arg file.
This is still in beta though: errors are being parsed from xcodebuild output, line by line. If error has multiline description, only the first line will be captured by Emcee. We are aiming to fix this in future releases of Emcee.
If you use xcodebuild with Emcee, you have to use insideUserLibrary location for simulators.
Emcee now has two new commands that allow you to disable specified worker, and to re-enable it later. This is useful for maintenance purposes.
$ Emcee disableWorker --worker-id "worker-machine-01" --queue-server "queue-host:41000"
$ Emcee enableWorker --worker-id "worker-machine-01" --queue-server "queue-host:41000"
There is also a corresponding REST handlers which queue server accepts.
Package.swift for highly modularized project quickly becomes a mess. Now this file is generated when you invoke make open and make gen. New changes to it's contents should be applied to Package.swift.template.
can finally be used to develop Emcee.
After performing test discovery, Emcee issues a request to store the obtained result into remote cache. Previously it didn't wait for it to complete. Sometimes this led to request not being completed. Now Emcee waits for request to finish.