Azkaban workflow manager.
This release has major changes related to containerization for the flow execution. It provides a way to isolate flow execution on containerized infrastructure like Kubernetes. For more detail, please refer to this design doc: https://azkaban.readthedocs.io/en/latest/containerization-design.html
9018fa2c (HEAD -> master, tag: 4.0.0, origin/master, origin/HEAD) Flow deterministic ramp-ups for imagetype and containerization 17bb970c Added jobtype based filtering for containerization ramp-up 0ae21638 Adding dependency for javax.el as it needs to be upgraded to 3.0 for hibernateValidator dependency (#2792) adce7491 Allow multiple copies of the same dependency file in Thin Zips. (#2754) 57286fb5 Merge Flow and server logs for containerized execution. (#2790) d1a6efe6 ApiGateway support for ramp-up mechanism from POLL to CONTANERIZED dispatch b285244c Give job props precedence over cluster props (#2786) 2ab81c19 Added mechanism to rampup Containerized from POLL dispatch 566bb1d8 Resolve symbolic links for jar files added to HadoopSecurityManagerClassloader (#2782) 7750f80f Marking flow as PREPARING from DISPATCHING as POD creation request is submitted (#2784) ab50ddcf Adding an option to provide flow container resource requests from flow parameter (#2780) 270defa2 Changed start status to READY instead of PREPARING for poll based dispatch 8024fada Add probes from the podTemplate to the podSpec e10a0226 Merge cf-engine-k8s to master (#2772) f49d8654 create-all-sql-*.sql is missing in some distributions 40625b17 Fix hyperlink to Containerization Design doc. Missed one leading "`" (#2767) 09e72927 Add Image Management APIs to index of documentation for easier access. (#2764) de5aa520 Updated image management API documentation. (#2766) 4a12ddec Delete unused "schedule options" files (#2755) c2b1634f Marking flow as PREPARING from DISPATCHING as POD creation request is submitted. (#2759) a9a8b8f0 Fix handling of null executionId in ExecutorApiGateway (#2752) e6febcc2 Fetching all the hadoop tokens from single method and moving call to specific method as responsibility of each implementation (#2746) 8d219eee Fixed image link for flowchart in imageMgmtAPI.rst (#2745) 902c6bd0 Containerization Design Doc (#2727) 16a6fd5b Bring back "Flow override properties to also override job properties" (#2632) f6678214 Add pipeline Level 3: block all jobs until the previous flow has completed (#2515) 7d770b2a Flow container cherry-pick from cf-engine-k8s (#2742) 87b1d561 Build Azkaban against Hadoop 2.10. (#2729) fb742825 Add executor api calls for containerized flows (#2715) b3c5bb63 Rename polling DB lock to include schema name (#2720) ce35c92f Add secret mounting to kubernetes dispatch (#2711) e3d8e5be Prevent kill action from leaving executions stuck in KILLING status. (#2691) 635a3fcc Restructuring code for HdfsAuth so that there can be multiple implementations of HdfsAuth and we can have an option to choose any implementation. (#2710) 670be395 Add ability to mount secrets in the PodSpec builder (#2702) cc46cd3d Updated AzKubernetesV1SpecBuilder.java to add hostPath volumes (#2689) 8c7926ce Add a date-time utility method and do a few refactors. (#2688) e8b4b6a9 Moving common HadoopSecurityManager code to AbstractHadoopSecurityManager so that all the versions of HadoopSecurityManager can reuse this code. (#2685) 51fe9b24 Solidified FlowContainer logic with integration testing. (#2687) 7a18f7bd Add TLS support to ExecutorApiGateway (#2679) 438930b7 Expose Azkaban Executor Server configuration to the cluster router (#2675) 4c9993fd Fixing the name of Property class for Containerized Dispatch. Earlier it was ContainerizedExecutionManagerProperties. (#2680) 1d68caff Added version_set table DDL and JdbcVersionSetLoaderImpl class for interaction 09154223 Add periodic cleanup of containers (#2671) 5ca7d39f It contains integration with service spec builder and service creation. (#2674) a10ff5d4 Added class for creation of kubernetes service object using template (#2667) 1c0642c6 It contains integration with pod spec builder and pod creation. It also contains delete methods for pod and service. (#2669) 6941dad2 Use CredentialProvider instead of CredentialProviderWithKeyStore unless needed. (#2664) 259b2510 Remove duplication in embedded flow name. (#2661) f7036b61 Job specific cluster components dependencies should take precedence over those of jobtype plugin. (#2660) e33546d5 BDP-3878: Magnet on flow ramping regardless failure (#2652) 65b25bfc Introducing version set and added skeleton for version set loader d41c9880 add a timer metric for flow startup delay (#2650) 7f86e683 Added builder class for creation of Kubernetes pod-spec 1121c83e Introducing FlowContainer. (#2647) 543797fb Merge robin branch into master (#2644) 1e8c84c1 Adding dispatch logic for containerized environment (#2635) 15222785 Expand the length of some fields related to Ramp Feature (#2630) b3c26fb4 Add a new CredentialProviderWithKeyStore interface which extends CredentialProvider interface with KeyStore specific APIs. (#2629) 8fb2667b Updaing dependency jopt-simple to 5.0.3 d0b519cb Add get/set KeyStore methods in CredentialProvider interface. (#2625) aee22b0a Removing duplicate code in ExecutorManager and ExecutionController (#2623) 8b65539a Fixed the PDT-->PST time change tests. They should pass until atleast 2029 (#2626) 5375629a Make MetricsManager more robust to changes in metrics reporter Ctor 43177fae Fix issue 917 and other startup issues 94f9ac84 Revert Flow Override feature #2241 (#2620) b60eb055 Patch6 (#2616) e9f17894 Add CSRF token validation for modifying project permissions f5e294d2 Fix startup errors in some environments (#2614) b2173ea1 Adding utility method to check PUSH and PUSH_CONTAINERIZED dispatch m… (#2607) c05f2339 Add flow execution source to flow properties (#2591) 3d07ead0 Add metric to measure the executor polling frequency. (#2599) 4386e776 Fixes failing WebUtils unit test (#2612) fae5b513 Update Travis CI configs. (#2609) 55b8eb9c Fix grammatical errors (#2605) 603afd4b Fix grammatical errors (#2606) b0aad21a Enable security logging via AzkabanEventReporter (#2608) 98eb6ca1 Restore flow lock message after project upload. (#2600) 9a50b73a Fix flaky queue duration condition in JobRunnerTest (#2604) 734121f6 Fix grammatical errors (#2601) f8a72d08 Fix grammatical errors (#2602) 783f88d4 Adding enum for Dispatch Method. It can be PUSH, POLL or CONTAINERIZED (#2603) 0fc132fa Formatting only: apply Save Actions on some files (#2596) ad72f4ac Flow override properties to also override job properties (#2241) 5b8eb08f Revert "Bring back Multiple Hadoop cluster (Robin) changes and fix LocalFileSystem leak. (#2585)" (#2592) 76537a35 Add job props to uniquely identify a job in mapreduce tags (#2593) 457957ca [UI] Improve readability of long job property values (#2580) 953b8729 Logging improvements for job callbacks (#2583) efef0034 Fix typos (quiting -> quitting) (#2581) 497d4cdb Add metrics to API endpoint requests. (#2590) 2217f85d fix bug in calculating flow kill and pause durations,and job kill and preparation durations (#2578) c79c8ce8 Remove null from startup-dependencies.json while writing and reading … (#2588) 1a78e1c1 fix typo for couple documents (#2584) dd075323 Bring back Multiple Hadoop cluster (Robin) changes and fix LocalFileSystem leak. (#2585) 99daa7b5 UI and API changes to prepare Azkaban for OAuth SSO. (#2587) 991238b3 Revert "Add metrics to API endpoint requests" (#2582) a041bcdc (origin/limit_flow_retry_count) Revert "Pass cluster object around instead of cluster id during job parameter construction in JobTypeManager." (#2579) ece3f081 Support sla settings in execute & schedule API actions (#2523) d18bdc83 Formatting only: apply Save Actions on some files (#2562) f2e3b681 Refactoring only: extract SLA parsing to new SlaRequestUtils (#2522) 31bd7422 Add metric NumAgedQueuedFlows: Count flows that are in queue for a long time (#2549) 23430813 Fixed the thread-pool usage when downloading dependencies 31fa0b76 Pass cluster object around instead of cluster id during job parameter construction in JobTypeManager. 604b3791 Add metrics to API endpoint requests 51eac654 Fix handling of Api failures during executor healthcheck 278a3135 Closed the IO streams in the thinzip code path. 1eec4e47 Expose cluster routing exceptions to job logger for easy debugging. 7e3899e1 Migrated to PDL ec24817f Fix jobrunner fireEvent timing. 5402abf8 Support routing jobs to a single Hadoop Cluster in Azkaban Exec Server. c4a16c7c Introduce a per-job classloader during job execution. 6e3635c7 Revert "Introduce a per-job classloader during job execution." 5ca11009 Modify fields for Azkaban Event Reporter. 9dc44864 Add Azkaban project event data. ee385ed8 Make HadoopSecurityManager_H_2_0 non-singleton. 519395c2 Introduce a per-job classloader during job execution. 48bc24f3 Revert "Introduce a per-job classloader during job execution." 3bd23cfc Introduce a per-job classloader during job execution. 687bb999 Remove Teradata jobtype in Reportal, and add idea plugin to build.gradle 5669e77d Changed assertEquals orders and remove a unit test assertion to prevent build error. 6253a370 added attributes for job and flow life-cycle events 4c296d12 Allow jobtype plugins to process job properties before jobs start to run. 2466fbb5 [UI] Correctly indicate when a schedule has SLA rules defined in the Schedules page. 634e202a Extract API endpoint names and some other (web|exec) server configurations into constants. 84dd461c Introduce a concept of cluster component, through which jobs and jobtypes can declare their dependencies. 5a0f1027 Change to Builder Pattern 1a98b4e4 Project Cache Abstraction 9255ef24 Handle exception during executor shutdown due to flow ramp disabled on clusters dc12ebac Remove direct instantiation and cast to Hdfs DistributedFileSystem dfc71f1a Improved error messages. Changed annotation for NotNull to Nonnull 98664650 Create an API to modify polling interval of an executor 9edab54d Refactor of Web server metrics. 183baf2c Add thread names for multi-threaded pools f9d71716 Upgrade Azkaban to the ~latest version of rest.li.
ab7aed47 (HEAD -> master, tag: 3.73.1, origin/master, origin/HEAD) Make UserUtils.setupWatch public. (#2261)
8ab89b94 (HEAD -> master, tag: 3.73.0, origin/master, origin/HEAD) Fix XmlUserManager bug: add back the mandatory constructor (#2258) f2daeeb0 Fix flaky FileWatcherTest (#2259) 505d2847 Auto expand RUNNING jobs (like FAILED and KILLED) in Flow Execution page, Job List tab. (#2257) 34b259e5 Fix HDFS viewer authentication issue. (#2255) 506a5fd0 FileWatcherTest fix (#2256) 05b71e44 Add logs to ProjectManager initialization at web server startup. (#2251) 320f3c09 XmlUserManagerTest to use mocks & avoid sleeping (#2244) d69a0d12 Job callback to support multiple occurrences of the same token (#2237) 44d8153a Fix NPE in dynamically generating job link url. (#2250) f0d6e0d0 Set maxParallelForks = 1 again in build.gradle (#2242) 054a54f9 Delete schedule of renamed/deleted flows on project upload. (#2226) 50f93f7e Add logging for the case when project cache still has enough buffer (#2239) 70ed1ea9 Add links to job logs in job context menu in execution graph (#2231) 11a1a563 Rename job logs link: Details -> Log (#2232) 2db6a8ba Update CONTRIBUTING.md (#2214)
Cherry-pick below PRs on top of 3.70.1: Fix dynamic job link url (#2189) Add web server host to event metadata (#2175) use fully qualified job name for azkaban.link.attempt.url and azkaban.link.jobexec.url (#2219)
1c61d83a [BUGFIX] Backward Imcompatible to some JobTypes (#2228) 2e371133 Implement ability to check executor's capacity before polling for new Dispatching Logic (#2195) 137ba43a revert flowid capitalization in ProjectManagerServlet.java (#2229) 38a6fca8 Enable dynamic job link for ExecutionController. (#2227) 04554187 WatchService : Make sure to wait for 1 second before acting on event to avoid multiple reloads (#2218)
d258ea7d Add APIs for locking a flow (#2210) 3308072d make logging explicit when fetching tokens from other namenodes (#2217) 18ec507f use fully qualified job name for azkaban.link.attempt.url and azkaban.link.jobexec.url (#2219) 4f30ad3b [DEDUPE] Refactor and Deduplicate the common codes (remerged) (#2186) 80cdc1a1 Enable web metrics for new dispatching logic. (#2213) 2852010c Add directory name to exception message when failing to create temp prop file (#2208) 21c555bf Add whitelist for specifying max number of concurrent runs for flows (#2201) d56d7cd7 fixed SessionfindByIp TestCase (#2188) d37cc8d7 Remove redundant UNIQUE INDEX declarations in CREATE TABLE statements (#2193)
cad96718 Fix dynamic job link url (#2189)
b2212b95 AUto Reload Config : Fix race condition to make sure WatchService is started before its watcher thread starts. (#2184) d9911be8 getProjectId api no longer returns IDs of deleted/inactive projects. (#2180) ae5aff4d When flow trigger fails to be scheduled/unscheduled, throw exception to terminate the subsequent steps (#2174) 53f30e4e Auto reload azkaban-users.xml if there is any change in it. (#2172) 5e9e00ef Reduce execution dir retention to 1 min (#2182) ae546a74 overrideSuccessEmails and overrideFailureEmails have mixed up its values (#1734) da7f2b03 Session monitoring enhancement (#2168) 9f99d731 Add web server host to event metadata (#2175) e718b6e4 Refactoring for SlaOption and ExecutionOptions (#2121) 3309ba27 Change ExecutorManager to ExecutorManagerAdapter in Flow Trigger (#2171) c2b00f56 Quartz scheduler refactor (#2161) 64cf66ee Closes #1066. Fix HadoopSecurityManager_H_2_0 to use addDelegationTokens API instead of deprecated/private getDelegationToken (#2165) 4e212ebb Refactor time related utility methods (#2164)
3cda04e3 Revert "Optimize gradle build time (#2113)" (#2163) bbb9a730 JobRunner: Quote JVM args (#2156)