Orchestrator Versions Save

Changes since v.3.0.13: https://github.com/github/orchestrator/compare/v3.0.13...v3.0.14

Some three months since previous release with much going on. Notable changes:

• New config: PreventCrossDataCenterMasterFailover, #766 to ensures failovers only take place within same datacenter, if so desired.
• Tagging: tag, untag, search instances by tags via #664, see documentation at https://github.com/github/orchestrator/blob/master/docs/tags.md
• KV store auto populates master info if not exists: #549
• Multi-values replication thread state, #767, thanks @ggunson
• GTID: support fixing errant transaction by injecting empty transaction on master, #707 introduces -c gtid-errant-inject-empty command.
• GTID: fix to master-master, #672, #673
• Fix to authentication in orchestratorclient, #681 , thanks @cezmunsta
• orchestrator-client: Add credentials to environment variables, #675 , thanks @mateusduboli
• FigureClusterName: Avoid heavy queries if clusterHint is empty, #727 , thanks @sjmudd
• fix create-per-test missing default value, #692, thanks @MOON-CLJ
• Fix to picking replica on balanced version / binlog_format state at failover, #773 , thanks @yangeagle
• Fix to take-master in master-master topology, #734
• raft support for SetClusterAliasManualOverride #776 , thanks @MOON-CLJ
• Added force-master-takeover API, #745 , thanks @MOON-CLJ
• Add --depends 'jq >= 1.5' tp fpm builds #752 , thanks @tomkrouper
• Some mitigation for reset-master risk, #706 , #762
• Web interface global message, #733
• Updated Docker + build, #774
• Documentation updates

And much more...

$ sha256sum *
9d63b6c0db3805bc12408b001188701db2889355a70969d9f49d30b66cfd5a2d  orchestrator-3.0.14-1.x86_64.rpm
a7ef69fcc10d7ce80a414e5c1e28ebf286ca4478d33a290fbdadc4fdbdc5d0e2  orchestrator-3.0.14-linux-amd64.tar.gz
4146a0aa6dc2f122d137937fb70eaeb4510573f8991727c34c48333bb1644582  orchestrator-cli-3.0.14-1.x86_64.rpm
6d8a9c42769583ed6366b71840a718c8b035e6d58298f58681f4dc9a42d98868  orchestrator-cli_3.0.14_amd64.deb
3685339dc83067527af6a8db81a315d40c8427c205894c44a69aa105995d17df  orchestrator-client-3.0.14-1.x86_64.rpm
0c0e5d7aa77a83f12aeac0d1503fd82151a776579367314a1427e491f2167d4f  orchestrator-client_3.0.14_amd64.deb
14725ce9ef3a57c2050793af172ab3d14c3abb73442329afdd79e4a44de52c61  orchestrator_3.0.14_amd64.deb

Changes since last release: https://github.com/github/orchestrator/compare/v3.0.12...v3.0.13

TL;DR Much Oracle GTID support, and many other things.

• GTID: detecting and analyzing errant transactions. On Oracle GTID clusters, orchestrator will:
  • Identify if a replica has errant transactions and will indicate the GTID set for those errant transactions, see https://github.com/github/orchestrator/pull/607
  • Web UI will visually indicate a server running with errant transactions, see https://github.com/github/orchestrator/pull/607
  • Allow "fixing" of errant transactions via reset master operation. This will auto calculate the errant range, and will correctly set gtid_executed and gtid_purged. orchestrator will protect against running this "fix" on an intermediate master. See https://github.com/github/orchestrator/pull/617
• GTID failure prediction:
  • Predict replication error and prevent relocation (https://github.com/github/orchestrator/pull/661). Will not allow relocating server A below server B if A is known to fail replication GTID-wise. MySQL can tell after effect, and orchestrator can tell before effect. This protects against loss of relay logs (especially important for SQL_DELAYed servers) and waste of futile operation. Thanks @MarkLeith, @evanelias, @sjmudd !
• GTID operations:
  • Better concurrency control, considers postponed-functions, utilizes failover optimizations (https://github.com/github/orchestrator/pull/641)
• Support for ReplicationCredentialsQuery config (https://github.com/github/orchestrator/pull/603), which allows orchestrator to grab replication credentials per-cluster; these can be used for setting up replication on graceful takeover or on master-master setup. Thanks @cezmunsta, @igroene !
• Fix to Zk KV update (https://github.com/github/orchestrator/pull/620)
• Fix to analysis GROUP BY statement (https://github.com/github/orchestrator/pull/632)
• Much internal refactoring (https://github.com/github/orchestrator/pull/623)
• Fixed integration tests on 5.7 (#649, #610), thanks @bbuchalter !
• Added web UI button for disable/enable global recoveries (#646, #656), thanks @mateusduboli !
• Added which-broken-replicas to list replicas with errors (#660), thanks @cezmunsta !
• Added which-cluster-osc-running-replicas command (#663), thanks @cezmunsta !
• take-siblings now uses "smart" logic (https://github.com/github/orchestrator/pull/670)
• raft internal health reports support HTTP auth. Thanks @almeida-pythian !
• Added MySQLConnectionLifetime, thanks @maciej-dobrzanski !
• Updates to docs, thanks @bbuchalter !
• Fix non-zero exit code for redeploy-internal-db (#624), thanks @bbuchalter !
• Include stdout/stderr in CommandRun's returned error (#635), thanks @bbuchalter !
• Fix: manual recovery overrides globally-disabled recoveries (#665)
• More visibility in logs in applying post-recovery steps (#666)

...and more!

$ gsha256sum *
2ca8f1d72ad1beae6048edc88efd3b0168593e42a2c6928e3cfa997d53062b79  orchestrator-3.0.13-1.x86_64.rpm
73ad98f312151c8aab5c708b940139fb8aa848c5891d096cf46ea83ce397bbe9  orchestrator-3.0.13-linux-amd64.tar.gz
9ceb60d669c2509f8d8d81f34591edcaafacf777606a995f742afcb1e40d3369  orchestrator-cli-3.0.13-1.x86_64.rpm
7b6f03b0b724e91d128b831ade51c1936efb847f0e39b9e749e370d59d631d7a  orchestrator-cli_3.0.13_amd64.deb
9192acf91ae0ba795918e20799fc6790af9275a3f25af73515e86cc5587dd826  orchestrator-client-3.0.13-1.x86_64.rpm
f2c13d0f679e8ef02216e2df697841f1ae4267a697c724901c0e5c5e09afa036  orchestrator-client_3.0.13_amd64.deb
26df8b44fa8bb6187f6b512a8fe4a63324b1faab006ec0c8959198a7b2e99427  orchestrator_3.0.13_amd64.deb

Changes since last release: https://github.com/github/orchestrator/compare/v3.0.11...v3.0.12

• New analysis: UnreachableMasterWithLaggingReplicas, handles situations where the master is in lockdown and replicas still think everything is fine. Handles by restarting IO thread on replicas, #572
• Massive reduction and redaction of logs, #555
• GTID-based promotion utilize immediate master promotion optimization, previously designed for Pseudo-GTID based failovers, #551
• Docker builds now using current branch as opposed to pulling master, #550
• Docker image now smaller, thanks @rhoml, #545
• Fix to basic authentication in orchestrator-client, #581
• Graceful master takeover: fail when attempting to promote ignored host, #570
• Graceful master takeover: now works when master is read-only, #548
• Added FailMasterPromotionIfSQLThreadNotUpToDate configuration, thanks @samiahlroos, #534
• orchestrator-client uses incremental sleep to auto-recover brief leader loss, #527
• Fixed timestamp on sqlite in non UTC boxes, #591, thanks @ndelnano
• start-replica returns an error if replication does not kick in in timely manner, #590, thanks @cezmunsta
• Fixed take-master logic when server already is co-master, thanks @Pomyk, #576

There's much more, and there are outstanding PRs which I am yet to review/test. Thank you to the many contributors!

$ gsha256sum *
2fbe927d9ee725d80ff30ba85dba2139c33bde5007f388ede8f4a05ee00799c1  orchestrator-3.0.12-1.x86_64.rpm
918d6950616b2cf16db019c7af8c7dae223de7b7ff1e3e221cad34baf734b0bb  orchestrator-3.0.12-linux-amd64.tar.gz
de8e639636d9d7fe17062780d4ad67ae43a9e0d2b913c62bac347aeffcc57dd4  orchestrator-cli-3.0.12-1.x86_64.rpm
9252b0cd387ea090c01983cb59cb510ad9d725236f3ef6adabff9d3a81ae456a  orchestrator-cli_3.0.12_amd64.deb
8aeb69c35225e30692ea3c203955314b0571e36539af64c92c76b0c59c745b0a  orchestrator-client-3.0.12-1.x86_64.rpm
ec8558217103be7a2457aabc6663545757332192f1e71310b59bed3a784fea53  orchestrator-client_3.0.12_amd64.deb
653ae7b6e2edfffd50cced387ce6d60247ff287fa7a8866d54bceb88d1bcdca6  orchestrator_3.0.12_amd64.deb

Changes since last release: https://github.com/github/orchestrator/compare/v3.0.10...v3.0.11

Notable:

• Graceful master takeover:
  • Supports multiple replicas to the master. Supports indicating the explicit replica onto which promotion takes place (orchestrator-client -c graceful-master-takeover -alias mycluster -d designated.master.to.promote:3306), #470
  • Graceful and forced failover: better anlaysis including GTID fix #509 This fixes a couple issues with GTID-based graceful takeovers.
• Better visual indication for GTID (indicate if server is configured with GTID but uses MASTER_AUTO_POSITION=0), #493
• KV support for ZooKeeper, #501
• Better propagation of errors in raft operations
• Contribution by @MaxFedotov, ACL support conf Consul KV, #510
• Contribution by @dveeden: build and doc fixes, #486, #487
• Contribution by @Slach: actualize Vagrant and fix problems on orchestrator-agent api with sqlite backend, #445

$ sha256sum *
58136686a8fbf14444b43f0e8f1a04efb76a9169f31b70590df09786283dd6e9  orchestrator-3.0.11-1.x86_64.rpm
3d21c14bc68cec1dc9b8cfc0c282e7984d07eda860b4ef340a5cef6e5ca41c32  orchestrator-3.0.11-linux-amd64.tar.gz
4e25a0f7a1327774f4d64b955eccb3bae4bd459103b3388f29807348ecda404c  orchestrator-centos6-3.0.11-1.x86_64.rpm
9edcfaf827691b60283e30a70dfa7878d15e05ea815a53064b49062f31fcbb8e  orchestrator-cli-3.0.11-1.x86_64.rpm
d19614df666ebe1d37a4b636b3e22aaa97926636c8247b2e5ba1458091ff756b  orchestrator-cli-centos6-3.0.11-1.x86_64.rpm
c9a9eda30f1250b02dc1b79774c8909ebaf5dcdc4917ff9eccda72262d8935cd  orchestrator-cli_3.0.11_amd64.deb
d2eba023b2f2ce0d0a2b0a5aa6c9e9e4684452d2d7d54868fb6bf864f782ffca  orchestrator-client-3.0.11-1.x86_64.rpm
2a53b3a39e094898188d994d4a04a569fa1c6b2c3d2817eada30bce44d5aca40  orchestrator-client-centos6-3.0.11-1.x86_64.rpm
e96190c8c13a0244cdb902bbebe89b6925e6b1830ca8c312d8040db43fcbceb9  orchestrator-client_3.0.11_amd64.deb
c845c13c0d888aff18174441a5a49c4da8732af904a07add1bcbd42fb031c781  orchestrator_3.0.11_amd64.deb

Changes since last release: https://github.com/github/orchestrator/compare/v3.0.9...v3.0.10

Notable:

• Graceful master takeover: support for PreGracefulTakeoverProcesses, PostGracefulTakeoverProcesses (thanks @igroene)
• Failure analysis contains command hint, advertised to hooks, #442 (thanks @choadrocker)
• Support for 'ack-all-recoveries' command (e.g. orchestrator-client -c ack-all-recoveries)
• Smarter cluster alias evaluation (#462), closes #459
• Docs: scripting cheatsheet, https://github.com/github/orchestrator/blob/master/docs/script-samples.md
• Contribution by @sjmudd: limit more precisely when special binlog filtering is used (#465)

$ sha256sum *
3b63347808a1290aad93b09d1fbe87884c18110f32f635e5280d9c692216d75a  orchestrator-3.0.10-1.x86_64.rpm
3456d393f23568622ab32b497cb4d2a1ae3f5f33c4cb754e5ba0e85d7b0965b0  orchestrator-3.0.10-linux-amd64.tar.gz
21abbd0d3b0ec0bdfb1af8fc548c0b9776e123dd6666b228d2e5c46421240d1e  orchestrator-centos6-3.0.10-1.x86_64.rpm
f90520dc9c514d2353e3099a1cc090499893ebbacc22e828194728b2439b5a55  orchestrator-cli-3.0.10-1.x86_64.rpm
e088c104ae6884a3b6cc4e1b152dd8ce3d8452ac184cbd7e4a088a7643e769dd  orchestrator-cli-centos6-3.0.10-1.x86_64.rpm
ea1481a602ba88ad0d1f62635fdfe192598366d3b07a6b0f0d7504377ce296e6  orchestrator-cli_3.0.10_amd64.deb
250b44d29038cfc08d2ed686d3ca7e860fa4e40ea9d81358d0d2454c27215d55  orchestrator-client-3.0.10-1.x86_64.rpm
36097b7a62726b104c084df1fbebd0dfa28471fea177239556809d286fa68ba0  orchestrator-client-centos6-3.0.10-1.x86_64.rpm
33c1218c9105b2e01762bb038f20481416bc5eec413afe9c2a65751a3b4690e8  orchestrator-client_3.0.10_amd64.deb
1198445ee375c61d716e03ca813940b600d43cf6be47f0fbe62576a8268b4067  orchestrator_3.0.10_amd64.deb

Changes since last release: https://github.com/github/orchestrator/compare/v3.0.8...v3.0.9

Notable:

• Introducing HTTPAdvertise config, #430, useful for orchestrator/raft on kubernetes deployments.
• Improved support for GTID via gtid_mode, #425, #427, more to come.
• Solved concurrent circular relocation issue via #420
• orchestrator/raft: followers report health status back to the leader, #431 . This can pave the way for delegation of tasks from leader to followers.
• Reduced noise for UnreachableMaster, #436
• Reducing backend DB write load: #432
• Fixing a concurrency issue: #435

$ sha256sum *
f0e059661f64a5c56c752862de454a2ab00a9eb84df8124e84eaa608a45a1591  orchestrator-3.0.9-1.x86_64.rpm
d91f3390de8c9847f1ad7198ba3b363f11d873a439dc1b2f14effb6e9cc736d7  orchestrator-3.0.9-linux-amd64.tar.gz
a305af04487d62586276347466f3a57f1deb979c3b7bae0f8f3d897a74eb3277  orchestrator-centos6-3.0.9-1.x86_64.rpm
417d7264322baa23cfc6f85e2b4ed8c533b70d79bb9fa4dc4b2504f36db51b88  orchestrator-cli-3.0.9-1.x86_64.rpm
ab4c70f0208f9a9a0e0e842aee9016963caed0ced0e2c0e1d1ac8368c1343152  orchestrator-cli-centos6-3.0.9-1.x86_64.rpm
0ea78bda98747e6693a34e58fb6dac5b05eebc590999ac163a84939c89889022  orchestrator-cli_3.0.9_amd64.deb
510f47dbdbdc9af550864821773c328311506b1d45bb4e2176706ec9f7d7c678  orchestrator-client-3.0.9-1.x86_64.rpm
02ec4fea3ffe0ca80e2f455377b62d5f00ed28746e9974419453220b15a92d04  orchestrator-client-centos6-3.0.9-1.x86_64.rpm
df0f1e1e47a40b2ff93cc1021b713e8dbeae5715414e0f459bc0ee55927e27a7  orchestrator-client_3.0.9_amd64.deb
bccf436cc808b2d219bab786992dd5055c5705c7e727dce30d09590e9a48981f  orchestrator_3.0.9_amd64.deb

Changes since last release: https://github.com/github/orchestrator/compare/v3.0.7...v3.0.8

This release consists of bugfixes found in 3.0.7:

• web interface:
  • Showing error response when submitting empty acknowledgement comment
  • "... blocked due to a ... previous recovery" links to the exact blocking recovery, not to general cluster listing.
  • Fixed typo
• Fixes raft persistence of cluster_injected_pseudo_gtid
• Added missing last-pseudo-gtid command in orchestrator-client

$ sha256sum *.*
7ff3a142dcd42c1f8f272ae39248134ed26a291537e5a979571df33b0a555d7b  orchestrator-3.0.8-1.x86_64.rpm
0932adcac3b330cbc533f10e5c4db1e0280efcc01ff9fb754999b47ba9e7c9ca  orchestrator-3.0.8-linux-amd64.tar.gz
6c5df74832cd4256db4608dea889edd4e18684b6bdccdbf1960727a19486d269  orchestrator-centos6-3.0.8-1.x86_64.rpm
7093f7665ff971f7c4f00dedb7a9937d1e249149531a79ef7c7eabaffc29eb65  orchestrator-cli-3.0.8-1.x86_64.rpm
de0815390753488bb64d483d5396d8840134f2990b90aeb728d1c1a48173e0ef  orchestrator-cli-centos6-3.0.8-1.x86_64.rpm
6a5fb95b53609d60d5b449d77076b8fccd55afe0426387618149c6a0862f17d7  orchestrator-cli_3.0.8_amd64.deb
eaa4b6a0355da28f60c4c753158fae06694feff4721f759ef58d65f1f6ec863c  orchestrator-client-3.0.8-1.x86_64.rpm
3b15d55b0886834ff3c1f4d1c57f548bf0e9653739924ffbca9d91943d87f346  orchestrator-client-centos6-3.0.8-1.x86_64.rpm
4148d4a210c46dca0f6d48fc4e8b6f8dc88bddb05ae1019cbd123e1ec6d8f778  orchestrator-client_3.0.8_amd64.deb
fe3ebd80088167ee999ba37927e00076170ef9dd489dd3a9dd50eab9e0cfdd6b  orchestrator_3.0.8_amd64.deb

Changes since last release: https://github.com/github/orchestrator/compare/v3.0.6...v3.0.7

Notable:

• Raft: non-leader nodes reverse proxy web/API requests to leader, assuming they're part of the healthy raft group.
  • Introducing new health check, /api/raft-health, responding 200 OK when a node is healthy in the raft group (i.e. not isolated)
• Support super_read_only: https://github.com/github/orchestrator/pull/390 adds UseSuperReadOnly config variables (default false). When true, orchestrator will set super_read_only together with read_only, and in particular on master promotion.
• Support SSL for master graceful takeover if SSL is enables (#401) -- thank you @stankevich !
• Custom JS and CSS (any customizations will not be supported) (#404) -- thank you @berlincount !
• Better handling of failure analysis when all replicas are downtimed.
• KV store: breaking down master identify (in addition to existing entry) Four more key/value pairs are added per master:
  • hostname
  • port
  • IPv4, if available
  • IPv6, if available
• Fixed lost-in-recovery downtime duration
• Improved web UI for recovery auditing, plus bugfixes (#413 , #415)

$ sha256sum *.*
c1cfc91d8cbdb8affabec1d9ba299562d5cd7d33b32958087af71d6d5eb96d9e  orchestrator-3.0.7-1.x86_64.rpm
c61873063c373481f8025a38a8907485526462d88d28f31548e5e17dcd87e6f8  orchestrator-3.0.7-linux-amd64.tar.gz
022bc1ade26c03620382d9b8ea4e1f17b4c28d91704f70825d89439ad934359e  orchestrator-centos6-3.0.7-1.x86_64.rpm
f142324a86e6c0df0487905e154e58523f5be75cb11e5b2dc7a7ad144bfdffbe  orchestrator-cli-3.0.7-1.x86_64.rpm
d58fb35aa84f5b9864f95980efbe5052846fd46f5112718fb0b887a16d643237  orchestrator-cli-centos6-3.0.7-1.x86_64.rpm
8dbbf11bf5f669c011048f3234e43deda603c1b414ae04271d0cba124d40d365  orchestrator-cli_3.0.7_amd64.deb
d59659a3c1f30e5f47462365b99513b765ef12d187593d39711ac273943da91c  orchestrator-client-3.0.7-1.x86_64.rpm
7ec3b6771b1a490a00661472f657fd49e3c37fe8fb4d8f6f62c5cce554cc1dd3  orchestrator-client-centos6-3.0.7-1.x86_64.rpm
ba6aa6f2f36afa3f6e9e350704c48667a709483b0426c257a87d1a2f5d0ec50f  orchestrator-client_3.0.7_amd64.deb
7cebf944c500d18cc802bac7150defcebc6d6cdf4b0537be9dc55323a6ac0126  orchestrator_3.0.7_amd64.deb

Changes since last release: https://github.com/github/orchestrator/compare/v3.0.5...v3.0.6

Noteworthy:

Automated Pseudo-GTID injection

With this release you may instruct orchestrator to inject Pseudo-GTID for you, via the new"AutoPseudoGTID": true variable. You may then drop or ignore any other Pseudo-GTID related configuration. See Pseudo-GTID configuration

orchestrator will auto-inject Pseudo-GTID where allowed to, and will auto recognize the masters onto which it is supposed to inject Pseudo-GTID. Pseudo-GTID gives one powers much like GTID, but without making the commitment to GTID.

$ sha256sum *
12b220607943a7e8e8ab8950fc770eb8489cd2e6c6c6a7c93b56375a0b766ae0  orchestrator-3.0.6-1.x86_64.rpm
bc8f74b155e662817cdb9782576a7bb5ee710ba07f372ffb9a498a921cca7f6b  orchestrator-3.0.6-linux-amd64.tar.gz
4e138bc3e4cc49745a156c9cc44cd633acf19491e21d2644b6c7fb8052398ede  orchestrator-centos6-3.0.6-1.x86_64.rpm
73bdf67475389f87393caa6b1496ec883303894b80102cedefed47dd0db1b1f3  orchestrator-cli-3.0.6-1.x86_64.rpm
5f7ebfa4ba3ad7c656aceea831295a1a4aba03c566854284fcb3752d6242081a  orchestrator-cli-centos6-3.0.6-1.x86_64.rpm
6fb89dad49618149deadb3f7c3e625e364953384095f3df8a438c449f7e0c95d  orchestrator-cli_3.0.6_amd64.deb
9fa1515e52764323c56d8b24ae021b100f05162358a93ef618c433fdff7703ee  orchestrator-client-3.0.6-1.x86_64.rpm
e20281a1e6fa38fff4cc3ad20260bd9286e180c6b0cc82dab83a85be38c9f0d8  orchestrator-client-centos6-3.0.6-1.x86_64.rpm
5b090c03ee3d5140f8b6c2040a612a27d34586d5c33b4fec10b3f9ca1ef8c1bd  orchestrator-client_3.0.6_amd64.deb
9d8bca2cdc45e4f54da98db97a86f9f02f2c11fe0c20e7e5b5a144cb0b612c30  orchestrator_3.0.6_amd64.deb

Changes since last release: https://github.com/github/orchestrator/compare/v3.0.3...v3.0.5

Noteworthy:

• Faster failure detection. Recall that orchestrator uses a holistic approach where is needs agreement (consensus, if you will) from replicas that a master or an intermediate master are dead. orchestrator requires a single agreement to all; there was an implicit and unintentional delay previously that causes orchestrator to run two detection cycles.
  • As result, orchestrator will report more UnreachableMaster incidents, seeing that a single network glitch can cause a server probe error. Recall that UnreachableMaster is not an actionable alert; this does not translate to false positives, but to more non-actionable failure detections.
  • This is known to reduce detection time by some 5 seconds, depending on polling interval.
• Faster failure detection where packets are dropped. Dropped packets typically cause confusion as they are not rejected and TCP takes more time to recognize failure. orchestrator will keep a concurrent stopwatch such that a probe may be hanging or blocked, yet detection proceeds uninterrupted.
  • This is known to reduce detection time by 5 to 15 seconds on packet drop incidents.
• More aggressive failure detection interval (once per sec, considerate of long running failure detection queries)
• Faster master failure recoveries. When, during recovery, orchestrator identifies that it has picked the "ideal" replica: the one that orchestrator really wants to promote and is marked as candidate (aka prefer), orchestrator immediately proceeds to promoting it as master, and asynchronously attaches the rest of the replicas below promoted server. This is known to reduce recovery time by 5-6 seconds.
• Support for semi-sync enable/disable & visibility.
  • introduced orchestrator -c enable-semi-sync-master, orchestrator -c enable-semi-sync-replica, orchestrator -c disable-semi-sync-master, orchestrator -c disable-semi-sync-replica
• Complete removal of prepared statements. orchestrator now uses client-side query argument interpolation. This saves round trips as well as makes orchestrator queries more easily identifiable.
• A detection that is failed over is implicitly acknowledged. This means while next failover (on same host) is potentially blocked (as per anti flapping interval), detection itself is not.
• Rewrite of failure detection & recovery documentation.
• tabulated ascii topology, making it easier on scripting & automation
• Forcing, validating and documenting RaftDataDir
• Fixed vagrant networking configuration - thanks @gtowey!
• UI changes
• Fixed orchestrator/raft bug where completed failovers were marked as "failed" on other raft member nodes.

$ sha256sum *
7bdd8a264832ae700586893b1814a1f44d387eb3a4cae99494cc03adbdf90a60  orchestrator-3.0.5-1.x86_64.rpm
434e7843db13c89d29dd03c9be1d8971372643866c0cc00848d9d70996ba1eb9  orchestrator-3.0.5-linux-amd64.tar.gz
f010ddd846731ce13886e921b8bddc35b7ef783bb53884b25883bf246f1ef153  orchestrator-cli-3.0.5-1.x86_64.rpm
265275d96346f6d7b582612a284b409458a1b5bb08e152aceae0e439a2b7990c  orchestrator-cli_3.0.5_amd64.deb
c7b34b19e12f018e7287a6bfb801157c127f1acf35b1add76eb5a00cff5241c0  orchestrator-client-3.0.5-1.x86_64.rpm
50c292833bd150aceee549ec9e9052ff0fd05b71dc6382bd56cbbc75ce71b033  orchestrator-client_3.0.5_amd64.deb
c2c5aebe9d4aa90a4decfcfbf747b4b231488c02a33df2264a2c49bfe12224a5  orchestrator_3.0.5_amd64.deb
35c2204d3d274491855026fcea6321ba1f45ea245f69e235f7373b2b1b3d143d  orchestrator-3.0.5-1.x86_64.rpm
434268a0db23bae82d0e26acdcdf8ad8e3b157a95221ab193e150ff8ec97e711  orchestrator-cli-3.0.5-1.x86_64.rpm
b206c28c2df497dd7b573665d1ce06deeedebbec407fb17eef08309791603554  orchestrator-client-3.0.5-1.x86_64.rpm