REST job server for Apache Spark
Merge pull request #1283 from bsikander/244_upgrade / (Noorul Islam K M | a448f2d5) feat(python): Disable hive support from Python Spark Session / (Behroz Sikander | 66f61509) refactor(python): Make subprocess.py PEP8 complaint / (Behroz Sikander | 875407ad) fix(tests): Stop context cleanly / (Sikander | d53f5074) feat(integration-tests): Add subproject for integration tests (#1285) / (Nibooor | d10522c0) refactor(readme): Set stop-context-on-job-error=true (#1284) / (Behroz Sikander | 61598af8) jobserver(python): Enable secure communication / (Behroz Sikander | 31fd78c2) doc(webapi): Add swagger documentation for Jobserver WebApi (#1282) / (Nibooor | bf54e185) fix(ci): Update CI scripts to 2.4.4 / (Behroz Sikander | da9dfd46) feat(jobserver): Add missing C* dependency / (Behroz Sikander | d88b956b) feat(jobserver): Disable hive support from SparkSession / (Behroz Sikander | f44aa509) feat(jobserver): Upgrade spark to 2.4.4 / (Behroz Sikander | 1244df7c) fix(dependent-jars): Dependent-jar-uris for context config / (Valentina Glagoleva | 8911bb17) misc(webapi): Standardize response functions and message / (Valentina Glagoleva | b9de4e1c) refactor(webApi): Remove unused code / (Valentina Glagoleva | 25d25340) fix(WebApi): Respond only with status code instead of "OK" / (Valentina Glagoleva | b4f13d34) refactor(WebApi): Use StatusCode obj for response codes / (Valentina Glagoleva | 90daa985) ci(jobserver): Build failing due to missing public key / (Behroz Sikander | 7fbf03ef) Update README.md (#1276) / (pgouda89 | 07672b3c) refactor(cassandra): Deprecate Cassandra DAO (#1272) / (Valentina Glagoleva | 611f7e68) feat(jobserver): Add /binaries/<appName> endpoint (#1273) / (Nibooor | 808010a6) Merge pull request #1267 from valan4ik/dependent_jars / (Noorul Islam K M | 96623139) test(jobdaoactor): Add test for broken URI case / (Valentina Glagoleva | 02e3c1e7) doc(readme): Update outdated coverage test command (#1270) / (Valentina Glagoleva | 23084634) fix(cassandra): Fix CassandraDAO config path / (Robin Wolters | 1536a2cf) refactoring(tests): Remove duplicating genJob function / (Valentina Glagoleva | 6b4192f0) fix(python): Improve job validation / (Valentina Glagoleva | 78ec91ff) doc(jobserver): Dependency jars and job submit / (Valentina Glagoleva | 6b08de40) fix(jobserver): Fix DAOs and refactor JobInfo / (Valentina Glagoleva | 1916d439) misc(cassandra): Add small readme and settings section / (Valentina Glagoleva | 9fe16d92) refactor(jobserver): Remove JobFileDAO / (Valentina Glagoleva | 8eef173e) feat(jobdaoactor): Validate schema for URIs / (Valentina Glagoleva | 35d66b9b) fix(binarymanager): Fix exception message type for cp / (Valentina Glagoleva | b74268cf) feat(jobserver): Extend JobInfo to maintain cp BinaryInfo list / (Valentina Glagoleva | 3a12f1cd) feat(jobserver): Add URI BinaryType / (Valentina Glagoleva | a1a44b58) internal(jobserver): Compatibility code for already running jobs / (Valentina Glagoleva | 5c79c429) feat(jobserver): Extend dependent-jar-uri feature / (Valentina Glagoleva | 179f7944) test(jobmanager): Add test for dependent-jar-uris feature / (Valentina Glagoleva | 91bf8da9) refactor(jobserver): Refactor CombinedDAO / (Robin Wolters | d69f5d7f) feat(jobserver): Move ZK DAO timers to CombinedDAO / (Robin Wolters | 28ab843d) feat(jobserver): Make stop-context-on-job-error default for streaming jobs / (Behroz Sikander | e0685858) feat(jobserver): Add stop-context-on-job-error feature / (Behroz Sikander | a4ac6cad) Fixes and multi-jvm tests for split brain resolver (#1264) / (Valentina Glagoleva | 3fb2d94a) feat(webapi): check sjs actors liveness (#1196) / (Srivignesh | a71fe073) doc(readme): Remove SPARK_CLASSPATH as it's deprecated / (Valentina Glagoleva | 187b7b3a) misc(akka): Upgrade Akka version to 2.4.20 / (Valentina Glagoleva | 832be9ca) fix(tests): Disable akka logging / (Behroz Sikander | 7ab20903) fix(test): Fix flaky test in JobManagerActorSpec / (Behroz Sikander | bf3bbfed) fix(test): Stop autopilot to fix flakiness / (Behroz Sikander | 695d0ff3) fix(tests): Cleanly clear SparkEnv object / (Behroz Sikander | e9f1e24a) refactor(jobserver): Add space in log message / (Behroz Sikander | cea3edde) fix(docu): Add docu for console logging for tests / (Behroz Sikander | 0f186368) fix(tests): Speed up SessionJobSpec test / (Behroz Sikander | 1e0504a1) fix(sbt): Use evaluated instead of inputTaskValue / (Behroz Sikander | 5c27de69) fix(readme): Add force=true flag for delete context / (Behroz Sikander | 1f025722) feat(sbt): Move sbt to 1.2.6 / (Behroz Sikander | 546dff52) fix(test): Fix spark.driver.host to localhost / (Behroz Sikander | d9129722) refactor(jobserver): Remove outdated TODO / (Robin Wolters | 6cb68101) fix(akka): Subscribe SBR to cluster events in constructor / (Valentina Glagoleva | f44c81fb) fix(akka): SBR should exit with -1 without hooks / (Valentina Glagoleva | 103e043f) feat(api): Handle all throwables in loadAndValidateJob / (Robin Wolters | 345cc7cc) refactor(jobserver): Address SBT warnings / (Valentina Glagoleva | 9b1addb1) doc(jobserver): Mention jobserver HA in README.md / (Robin Wolters | c5fc2e17) doc(ha): Add documentation for HA setup / (Robin Wolters | 3433f048) doc(jobserver): Update DAO setup documentation / (Robin Wolters | 5533ded9) fix(jobserver): Decrease down-removal-margin / (Robin Wolters | c6a5e948) doc(splitbrain): Add documentation for Jobserver split brain resolver / (Valentina Glagoleva | 368125af) feat(akka): Add split brain resolver / (Valentina Glagoleva | 832fef96) fix(jobserver): Add watch to existing contexts automatically / (Robin Wolters | f6082df3) fix(jobserver): Don't allow STOPPING context to restart / (Valentina Glagoleva | 3e3151fe) feat(jobserver): Add HA setup for jobserver (#1241) / (Behroz Sikander | d7f26825) fix(jobserver): Fix custom exceptions constructor / (Valentina Glagoleva | f975d8b9) feat(jobserver): Make AkkaClusterSupervisorActor a singleton / (Robin Wolters | 9448ee75) Fixes related to StopContext flow (#1238) / (Nibooor | 8ef21212) doc(jobserver): Update jobs state machine (#1237) / (Behroz Sikander | 2b6c0acd) Only respond if cleanup is complete (#1233) / (Behroz Sikander | af63e846) refactor(webapi): Deprecate WebAPI /jars path / (Valentina Glagoleva | 16eaff51) feat(sbt): Use cached resolution / (Valentina Glagoleva | 7448975e) CassandraDAOSpec: Fix of flaky test and small refactoring (#1231) / (Valentina Glagoleva | 483840c8) feat(jobserver): Clean non-final jobs at startup / (Behroz Sikander | fa749b7d) fix(jobserver): Repair final states without end time / (Robin Wolters | 0afcc7e4) fix(jobserver): Properly set end_time on ForkedJVMInitTimeout / (Robin Wolters | 75954de7) refactor(jobserver): Improve log messages / (Behroz Sikander | 1a4e55e8) refactor(jobserver): Add logging before terminating JVM / (Behroz Sikander | 3780682b) refactor(jobserver): Explain CombinedDAO binary deletion error / (Robin Wolters | d978b9d5) fix(tests): Make NamedRDD calculation longer for timeout tests / (Valentina Glagoleva | f025a4c4) fix(tests): Add autodown to AkkaClusterSupervisorSpec / (Valentina Glagoleva | 5e995b50) refactor(tests): AkkaClusterSupervisor compliant with scalastyle / (Valentina Glagoleva | b4f7ad87) fix(tests): Cleanup SQL database in tests / (Valentina Glagoleva | 51aa2d8f) fix(jobserver): Persist error state sequentially during startup / (Robin Wolters | f4e1edd5) fix(jobserver): Remove refreshBinary for Zookeeper / (Robin Wolters | 302cf921) fix(tests): Adapt AutoPurgeActorSpec to slow builds / (Robin Wolters | 4e94910d) doc(jobserver): Add documentation of Zookeeper autopurge feature / (Robin Wolters | 9c84c976) feat(jobserver): Automatically purge Zookeeper data / (Robin Wolters | 5193eaa8) refactor(jobserver): Replace hard-coded dao timers with config values / (Robin Wolters | 94ed79bc) fix(jobserver): Add timers to ZookeeperDAO / (Robin Wolters | cc6a28a6) fix(jobserver): Add timers to WebAPI / (Robin Wolters | f543b00b) ZookeeperDAO fixes (#1216) / (Nibooor | 9b2b600d) doc(jobserver): Add context creation flow diagram (#1215) / (Nibooor | 206dec54) Minor fixes and improved logging (#1214) / (Behroz Sikander | b0542424) Remove binaries if not used & adding logging to webapi (#1209) / (Behroz Sikander | a2d5eb81) fix(tests): Fix saveJobConfig tests (#1210) / (Behroz Sikander | 489b45af) Make zookeeper settings configurable (#1211) / (Valentina Glagoleva | 53d335cb) refactor(jobserver): Remove blocking await from StartJob / (Valentina Glagoleva | d7057514) Downgrade zookeeper version and remove not needed ZK test helper (#1206) / (Valentina Glagoleva | f4f29ed0) feat(jobserver): Force consistent read with manual sync (#1205) / (Nibooor | 14a1d4f2) Minor refactor for WebApi logging (#1200) / (Valentina Glagoleva | dc703ad1) Merge pull request #1201 from valan4ik/combined_dao_doc_conn / (Noorul Islam K M | 399f7311) fix(travis): Update base container tag (#1204) / (Behroz Sikander | 6d3e6160) fix(zookeeper): Scale timeouts to avoid ConnectionLoss exception / (Valentina Glagoleva | 372d5104) doc(jobserver): Add documentation for CombinedDAO / (Valentina Glagoleva | 815ee6ac) fix(jobserver): Create Job with full BinaryInfo information / (Valentina Glagoleva | d339ce34) fix(jobserver): Return JobInfo with binHash in SQL DAOs / (Valentina Glagoleva | 1ad61b06) fix(jobserver): keeping unused binaries (#1198) / (Behroz Sikander | ae54f486) fix(jobserver): Exclude guava from curator dependencies / (Robin Wolters | ca20f24b) fix(tests): Fix build issues with test setup / (Robin Wolters | 4c70b9f7) refactor(jobserver) Improve ZooKeeper logging / (Robin Wolters | c4bba5c2) feat(jobserver): Reuse zookeeper connections within same dao operation / (Robin Wolters | 6c109472) fix(tests): Fix curator framework import and namespace / (Valentina Glagoleva | 4f4d6427) feat(jobserver): Implement ZookeeperDao / (Robin Wolters | 61061c9f) feat(jobserver): Add ZookeeperDAO / (Valentina Glagoleva | fe80b438) feat(jobserver): Add ZK basic utils and tests / (Valentina Glagoleva | 21b9cb0d) feat(jobserver): Include curator and zk dependencies / (Behroz Sikander | d36d1321) feat(akka-app): Expose mean duration of timers in metricz API endpoint / (Robin Wolters | c29a03e9) fix(jobserver): Fix unlimited cluster.join request / (Robin Wolters | 706cd36e) fix(jobserver): Align GetSparkContexData query timeouts / (Robin Wolters | 67bf1d7e) fix(jobserver): ensure job cleanup after fatal error - ref #1161 / (Sébastien Gautrin | b361c079) Adapt test configurations to run on slow build system (#1184) / (Behroz Sikander | a0a28fbc) feat(jobserver): Add counters for API requests / (Robin Wolters | 3e1a59cf) fix(jobserver): Remove concurrency for save operations in tests / (Valentina Glagoleva | 2327e44f) fix(jobserver): Use random HDFS port for HDFS tests / (Valentina Glagoleva | a066c08a) Merge pull request #1170 from bsikander/cassandra_connector / (Noorul Islam K M | be00f0ef) Merge pull request #1172 from bsikander/cd_fixes / (Noorul Islam K M | 501cb75c) fix(jobserver): Increment counter if other app is using bin / (Behroz Sikander | 183827dd) feat(jobserver): Add metrics for combined dao / (Behroz Sikander | 5820e700) fix(jobserver): Do not convert stream to character stream / (Behroz Sikander | 1f04503b) fix(jobsever): save context/job/config should be synchronous / (Behroz Sikander | 1b206aa5) fix(jobserver): Create rootdir folder / (Behroz Sikander | 5a075201) fix(jobserver): Add cache-on-upload feature in CombinedDao / (Behroz Sikander | ce01aa59) Merge pull request #1169 from valan4ik/split_dao / (Noorul Islam K M | de4754d6) feat(job-server): pass context name as appName when context is submitted / (i.kushmantsev | 9778ef8d) fix(jobserver): Minor changes for gitignore and scripts shebang / (Valentina Glagoleva | 18d18831) fix(jobserver): Upgrade C* connector version / (Behroz Sikander | ad300f8d) misc(jobserver): Used correct spark download URL (#1171) / (Noorul Islam K M | 00891e24) fix(jobserver): Fixes for CombinedDAO / (Valentina Glagoleva | dc1dff4d) fix(jobserver): Wait for futures to complete / (Valentina Glagoleva | 1bbda1d3) feat(jobserver): SQL implementation of MetaDataDAO / (Ecaterina Andreev | d65d03ea) feat(jobserver): add getBinariesByStorageId function to MetaDataDao / (Ecaterina Andreev | f80b89b8) fix(jobserver): Fix null pointer exception for logger / (Valentina Glagoleva | c02621f8) feat(jobserver): Add config validation for Combined/Hdfs DAOs / (Valentina Glagoleva | da174628) fix(jobserver): Fix resolving filesystem schema / (Valentina Glagoleva | 1f4d2d14) feat(jobserver): Add HDFS implementation for BinaryDAO / (Valentina Glagoleva | e18e132e) feat(jobserver): Add CombinedDAO class / (Valentina Glagoleva | add0532e) feat(jobserver): Introduce BinaryDAO and MetaDataDAO / (Valentina Glagoleva | 14441350)
feat(jobserver): Replace manager_start with SparkLauncher (@bsikander) - this is a pretty big deal to stabilize and extend SJS
feat(jobserver): Create context table (@ecandreev)
feat(jobserver): move data from contexts hash to db (@ecandreev)
feat(jobserver): Add CONTEXT_ID and STATE to the JOBS table (@ecandreev)
feat(jobserver): Introduce supervise mode (@bsikander)
Updated Python documentation to reference PythonSession (@rfrohock)
fix(webapi): repair context info route for mode context-per-jvm (@pk-work)
feat(webapi): Add application id to route /contexts/
test(jobserver): Add testcases for AkkaClusterSupervisor (@bsikander)
fix(jobserver): Fix randomly failing testcase in AkkaClusterSupervisorSpec (@bsikander)
feat(jobserver): Allow valid memory value and add testcases (@bsikander)
feat(jobserver): Remove extra states from contextInitInfos (@bsikander)
feat(jobserver): Extend zombie killing logic to add Initialize msg (@bsikander)
fix(jobserver): Fix testcases which simulate DAO failure (@bsikander)
feat(jobserver): Remove extra states from contextInitInfos (@bsikander)
Improved documentation in readme for H2 db (@ericraymond)
docs(jobserver): Add new logging doc and a line to cluster doc (@LXXIII73)
docs(jobserver): Update JMX documentation for Akka Cluster (@bsikander)
fix(job-server): use 24h format for cached binary file name (@sgautrin)
docs(jobserver): Fix broken links (@VEINHORN)