The simple, stupid batch framework for Java
This is a patch release that fixes the following issue:
FileRecordReader
This release also includes some minor backward-compatible dependency updates:
2.12.0 -> 2.12.2
1.0.7 -> 1.0.8
2.9.0 ->2.9.1
1.4.15 -> 1.4.16
5.3.2 -> 5.3.4
5.4.26.Final -> 5.4.29.Final
This is a patch release that fixes the following issue:
This release also includes some minor backward-compatible dependency updates:
2.11.2 -> 2.12.0
2.8.4 ->2.9.0
1.4.12 -> 1.4.15
1.13 -> 1.15
5.2.9.RELEASE -> 5.3.2
5.4.21.Final -> 5.4.26.Final
6.1.5.Final -> 6.1.7.Final
This release is a major version as it requires Java 11 or above. The main theme for this release is API type safety. There are no new features in this version except updating all APIs to be generic:
RecordReader (and its listener): 7ed62a5d3c0a3ddfaa89fbc0757fddd6c5591288Batch (and its listener): ba60977cd2135e3f91968ad2f95744d474e77bdcRecordWriter (and its listener): d437aa550f0c9654fde355f68261b3efd98c0aa4RecordProcessor (and its listener / sub interfaces): 279ed297d5eb60d590958a11d8971f70199d6e5fPredicate: 063fcc573d5783f48c01d3375e6d6a6459bf5d4eJobBuilder: 69299364256fade74a219553540bf04222ef62d8These changes require updating job definitions to use generics for input/output types in order to enforce the correctness and coherence of expected types between the record reader and writer (More details about this in issue #388).
The most notable change required by this release is regarding job definitions:
--Job job = new JobBuilder() // v6
++Job job = new JobBuilder<String, Tweet>() // v7
// define batch components
.build();
Specifying input/output types is not mandatory, but not doing so will lead to a raw type usage warning.
Another important change has been introduced in this release about the generic type of RecordProcessor. In v6, the generic types of RecordProcessor were the types of input/output records. In v7, generic types now represent the type of the payload of input/output records:
--public interface RecordProcessor<I extends Record, O extends Record> { // v6
-- O processRecord(I record) throws Exception;
--}
++public interface RecordProcessor<I, O> { // v7
++ Record<O> processRecord(Record<I> record) throws Exception;
++}
This change has a direct impact on all interfaces extending RecordProcessor (like RecordFilter, RecordMapper, RecordMarshaller and RecordValidator) and their corresponding implementations.
Any usage of the aforementioned APIs (RecordReader, RecordWriter, Batch, etc) and their associated listeners should be updated to use generics where appropriate.
All deprecated APIs in v6.0 and v6.1 have been removed in this major release. The suggested replacement (if any) should be found in the Javdoc of the deprecated API.
This is a minor release which can be used as a drop-in replacement for v6.0.0. Here are the major changes:
RetryableRecordProcessor implementationBeanPropertiesPreparedStatementProvider is failing to prepare statement if bean properties has null valueJobBuilder#aNewJob
ContentBasedBlockingQueueRecordWriterBuilder
This major release marks a new generation of the framework which is now based on Java 8. It has been an opportunity to improve the framework internals as well as some public APIs. All deprecated APIs in v5.3 and before have been removed. Some APIs have changed in a non-backward compatible way to fix a couple of minor design inconsistencies introduced in the v5 line.
The root package has been updated from org.easybatch to org.jeasy.batch for consistency with other Jeasy projects. Artifact IDs have also been changed to match the same naming pattern as other projects. Please refer to the migration guide below for more details.
As you will see in the migration guide, many APIs have been deprecated in previous versions and are now completely removed. As Antoine de Saint-Exupery said:
"Perfection is achieved not when there is nothing more to add, but when there is nothing left to take away"
I believe with this release, there is no much more to take away from Easy Batch, but this does not make it perfect! That said, I am very happy and proud to see some serious companies using it in production as well as encouraging feedback from the community. So I would like to thank all contributors who helped making this release possible! Thank you all for your time and efforts! You are awesome 👍
In addition to dependencies and documentation updates, here are the major highlights for v6.0:
JobExecutor implement java.lang.AutoCloseable
BeanFieldExtractor
FileRecordWriter should not throw an Exception at construction timeJobReport#toString
XmlRecordReader should close the input streamjava.io.File (namely in constructors of all file based record readers/writers) were deprecated in v5.3 and have been replaced with java.nio.file.Path in v6easybatch-quartz has been removed in v6. This module used to provide two classes that act as a bridge between Easy Batch APIs and Quartz APIs. Those two classes (EasyBatchJob and EasyBatchJobFactory) can now be copied from the Quartz tutorial.@Deprecated annotations on classes of this module in the v5.3 release. Sorry for any inconvenience!JobListener#beforeJobStart and JobListener#afterJobEnd have been renamed to JobListener#beforeJob and JobListener#afterJob (Issue #362)Destination instead of Queue to support Topics as well (Issue #359)JmsMessageTransformer now accepts a javax.jms.Session instead of javax.jms.QueueSession to support both queue and topic sessions (Issue #360)JmsRecordReader now returns null based on a timeout and not a JmsPoisonMessage anymore (because JmsPoisonMessage has been removed, see "Removed APIs" section of the migration guide)BlockingQueueRecordReader now returns null based on a timeout and not a PoisonRecord anymore (because PoisonRecord has been removed, see "Removed APIs" section of the migration guide)ContentBasedBlockingQueueRecordWriter, ContentBasedBlockingQueueRecordWriterBuilder, DefaultPredicate, Predicate, RandomBlockingQueueRecordWriter, RoundRobinBlockingQueueRecordWriter) and their JMS equivalent were moved to a new extension module called easy-batch-integration
Although care has been taken to document all changes in details, a thing or two could have been missed (in which case, apologies upfront).
org.easybatch to org.jeasy
easybatch-core -> easy-batch-core (same pattern for other artifacts)Here is the new maven dependency for the core module:
<dependency>
<groupId>org.jeasy</groupId>
<artifactId>easy-batch-core</artifactId>
<version>6.0.0</version>
</dependency>
org.easybatch.xml.XmlWrapperTagWriter was deprecated in v5.3 and has been removed in v6org.easybatch.tools.monitoring.CliJobMonitoringListener, org.easybatch.tools.reporting.HtmlJobReportFormatter andorg.easybatch.tools.reporting.JobReportEmailSender were deprecated in v5.3 and have been removed in v6. The entire easybatch-tools module has been removedeasybatch-quartz has been removed in v6@Deprecated annotation on classes of this module in the v5.3 release. Sorry for any inconvenience!org.easybatch.jms.JmsPoisonMessage, org.easybatch.jms.JmsPoisonRecord, org.easybatch.jms.JmsPoisonRecordBroadcaster and org.easybatch.jms.JmsPoisonRecordFilter were deprecated in v5.3 and have been removed in v6.org.easybatch.core.record.PoisonRecord, org.easybatch.core.listener.PoisonRecordBroadcaster and org.easybatch.core.filter.PoisonRecordFilter were deprecated in v5.3 and have been removed in v6.org.jeasy.batch.core.reader.BlockingQueueRecordReader#BlockingQueueRecordReader(java.util.concurrent.BlockingQueue<org.jeasy.batch.core.record.Record>, int) has been removed in v6. Please note that this constructor was not marked with @Deprecated in v5.3 by mistake.BlockingQueueRecordWriter(final List<BlockingQueue<Record>> has been be removed in v6 as this writer now operates on a single blocking queue instead of multiple queues.org.easybatch.core.listener.RecordProcessingTimeListener was deprecated in v5.3 and has been removed in v6org.easybatch.core.filter.RecordNumberBetweenFilter, org.easybatch.core.filter.RecordNumberEqualToFilter, org.easybatch.core.filter.RecordNumberGreaterThanFilter and org.easybatch.core.filter.RecordNumberLowerThanFilter were deprecated in v5.3 and have been removed in v6JobReportFormatter and DefaultJobReportFormatter were deprecated in v5.3 and have been removed in v6DelimitedRecordMarshaller and ApacheCommonCsvRecordMarshaller have been removed. Use the new setters for these parameters instead.org.easybatch.core.filter.FilteredRecordsSavingRecordFilter was deprecated in v5.3 and has been removed in v6. Use the new org.easybatch.core.filter.FilteredRecordsCollector insteadorg.easybatch.core.filter.StartWithStringRecordFilter was deprecated in v5.3 and has been removed in v6. Use org.easybatch.core.filter.StartsWithStringRecordFilter insteadorg.easybatch.core.filter.EmptyRecordFilter was deprecated in v5.3 and will be removed in v6. Use org.easybatch.core.filter.EmptyStringRecordFilter insteadorg.easybatch.core.filter.EndWithStringRecordFilter was deprecated in v5.3 and will be removed in v6. Use org.easybatch.core.filter.EndsWithStringRecordFilter insteadjava.util.date (namely in Header class) were deprecated in v5.3 and have been replaced with java.time.LocalDateTime in v6java.io.File (namely in constructors of all file based record readers/writers) were deprecated in v5.3 and have been replaced with java.nio.file.Path in v6PayloadExtractor was deprecated in v5.3 and has been removed in v6. Use Utils#extractPayloads insteadorg.jeasy.batch.extensions.yaml.YamlRecordReader#YamlRecordReader(java.io.InputStream, java.lang.String) has been replaced with one that takes a java.nio.charset.Charset instead of a String for the charset name. Please note that this constructor was not marked with @Deprecated in v5.3 by mistake.org.jeasy.batch.json.JsonRecordReader#JsonRecordReader(java.io.InputStream, java.lang.String) has been replaced with one that takes a java.nio.charset.Charset instead of a String for the charset name. Please note that this constructor was not marked with @Deprecated in v5.3 by mistake.org.easybatch.extensions.stream.StreamRecordReader was moved to the core module under org.easybatch.core.reader
JobMonitor and JobMonitorMBean) have been moved from org.jeasy.batch.core.job to org.jeasy.batch.core.jmx
JpaRecordReader#setFetchSize has been renamed to JpaRecordReader#setMaxResults. Note that this has not been deprecated in v5.3.This is a minor release which is the last release of the v5.x line. Here are the major changes:
FileRecordWriter
JobReportFormatter in JobReportEmailSender
JobMetrics merge loops in DefaultJobReportMerger
XMLStreamException
HibernateRecordWriter is rolled backJpaRecordWriter is rolled backJdbcRecordWriter is rolled backjava.util.Date) field in JdbcRecordWriter
HtmlJobReportFormatter should not generate apache velocity logsXmlRecordReader throws a ClassCastException when reading a comment lineorg.easybatch.xml.XmlWrapperTagWriter is deprecated in v5.3 and will be removed in v6org.easybatch.tools.monitoring.CliJobMonitoringListener, org.easybatch.tools.reporting.HtmlJobReportFormatter andorg.easybatch.tools.reporting.JobReportEmailSender are deprecated in v5.3 and will be removed in v6. The entire easybatch-tools module will be removedeasybatch-quartz module are deprecated and the entire module will be removed in v6@Deprecated annotation on classes of this module in the v5.3 release.org.easybatch.jms.JmsPoisonMessage, org.easybatch.jms.JmsPoisonRecord, org.easybatch.jms.JmsPoisonRecordBroadcaster and org.easybatch.jms.JmsPoisonRecordFilter are deprecated in v5.3 and will be removed in v6. The JmsRecordReader will return null based on a timeout and not a JmsPoisonMessage.org.easybatch.core.record.PoisonRecord, org.easybatch.core.listener.PoisonRecordBroadcaster and org.easybatch.core.filter.PoisonRecordFilter are deprecated in v5.3 and will be removed in v6. The org.jeasy.batch.core.reader.BlockingQueueRecordReader will return null based on a timeout and not a PoisonRecord
org.jeasy.batch.core.reader.BlockingQueueRecordReader#BlockingQueueRecordReader(java.util.concurrent.BlockingQueue<org.jeasy.batch.core.record.Record>, int) will be removed in v6. Please note that this constructor was not marked with @Deprecated in v5.3 by mistake.BlockingQueueRecordWriter(final List<BlockingQueue<Record>> will be removed in v6 as this writer will operate on a single blocking queue.org.easybatch.core.listener.RecordProcessingTimeListener is deprecated in v5.3 and will be removed in v6org.easybatch.core.filter.RecordNumberBetweenFilter, org.easybatch.core.filter.RecordNumberEqualToFilter, org.easybatch.core.filter.RecordNumberGreaterThanFilter and org.easybatch.core.filter.RecordNumberLowerThanFilter are deprecated in v5.3 and will be removed in v6JobReportFormatter and DefaultJobReportFormatter are deprecated in v5.3 and will be removed in v6org.easybatch.core.filter.FilteredRecordsSavingRecordFilter is deprecated in v5.3 and will be removed in v6. Use the new org.easybatch.core.filter.FilteredRecordsCollector insteadorg.easybatch.core.filter.StartWithStringRecordFilter is deprecated in v5.3 and will be removed in v6. Use org.easybatch.core.filter.StartsWithStringRecordFilter insteadorg.easybatch.core.filter.EmptyRecordFilter is deprecated in v5.3 and will be removed in v6. Use org.easybatch.core.filter.EmptyStringRecordFilter insteadorg.easybatch.core.filter.EndWithStringRecordFilter is deprecated in v5.3 and will be removed in v6. Use org.easybatch.core.filter.EndsWithStringRecordFilter insteadjava.util.date (namely in Header class) are deprecated and will be replaced with java.time.LocalDateTime starting from v6java.io.File (namely in constrcutors of all file based record readers/writers) are deprecated and will be replaced with java.nio.file.Path starting from v6PayloadExtractor is deprecated in v5.3 and will be removed in v6. Use Utils#extractPayloads insteadorg.jeasy.batch.extensions.yaml.YamlRecordReader#YamlRecordReader(java.io.InputStream, java.lang.String) will be replaced with one that takes a java.nio.charset.Charset instead of a String for the charset name. Please note that this constructor was not marked with @Deprecated in v5.3 by mistake.org.jeasy.batch.json.JsonRecordReader#JsonRecordReader(java.io.InputStream, java.lang.String) will be replaced with one that takes a java.nio.charset.Charset instead of a String for the charset name. Please note that this constructor was not marked with @Deprecated in v5.3 by mistake.For more details about deprecations and replacements, please refer to the Javadocs.
The next version will be v6 and will be based on Java 8. The root package name will be updated from org.easybatch to org.jeasy.batch for consistency with other Jeasy projects. Artifact IDs will also change to match the same naming pattern as other projects.
I would like to thank all contributors (@anarayn, @MALPI , @jcamiloradamesa , @IvanAtanasov, @sheikhu, @vian42, @verdi8 and @psoares-resilient) for submitting bug reports, testing fixes and contributing to the project with awesome PRs!
This is the second maintenance release of the v5.x line. It brings some features and bug fixes:
PipelineListener call orderFilteredRecordsSavingRecordFilter
XmlRecordReader does not escape gt and lt
XmlRecordReader does not call getLocalPart() on QName
DefaultJobReportFormatter.formatReport duration rolls over after 24 HoursFilteredCount metric should be renamed to FilterCount
file and charset in AbstractFileRecordReader
otherwise method in content based queue record writer builders should be optionalThe following methods are deprecated due to renaming FilteredCount to FilterCount in issue #296:
org.easybatch.core.job.JobMetrics#getFilteredCount
org.easybatch.core.job.JobMetrics#incrementFilteredCount
org.easybatch.core.job.JobMonitor#getFilteredCount
org.easybatch.core.job.JobMonitorMBean#getFilteredCount
These methods will be removed in v5.3.
I would like to thank @DanieleS82, @ipropper and @tobias-- for their time and effort to make this release happen!
This is minor release adding some new features and bug fixes. Here are the most important changes:
MultiFileRecordReader
XmlFileRecordReader
JsonFileRecordReader
XmlRecordReader
JsonRecordReader
JmsRecord headerJobReport
JdbcRecordReader
JsonFileRecordReader should not throw a FileNotFoundException at construction timeXmlFileRecordReader should not throw a FileNotFoundException at construction timeQueueRecordWriter builders to guide the user through predicate/queue mappingXmlFileRecordReader constructor parameters should be swappedJmsQueueSessionListener
JmsQueueConnectionListener
Many thanks to all contributors for reporting bugs and testing fixes. A special thank to @AussieGuy0 for adding support for uniVocity parsers!
This is a bug fix release for the following issue:
Many thanks to @packley1 for reporting the issue and testing the fix.
The main theme of this new major release is less is more :exclamation: With this new version, Easy Batch has never been easier! Many APIs have been simplified and improved for more consistency and ease of use. Some APIs have been removed in order for you to do less work and delegate more to the framework :smile:
After one year of active development, version 5.0.0 is finally here. This would not have been possible without the amazing open source community !! Many thanks to all contributors who suggested features, filed bugs and reported inconsistencies. I'm really grateful and thankful to all of you!
Version 5 is not 100% backward compatible with version 4. It introduces fundamental changes in the core framework in order to fix some inconsistent APIs. The processing workflow has been modified to improve performance.
This is the final release of v5.0 after many snapshot versions and 2 release candidates. Please find below the complete change log and a migration guide to help you move from v4 to v5.
Major changes
New features
RetryableRecordReader
RetryableRecordWriter
CompositeRecordWriter
Bug fixes
BeanValidationRecordValidator
BlockingQueueRecordReader / BlockingQueueRecordWriter
JobStatus.ABORTED is redundantXmlWrapperTagWriter
Enhancements and API changes
RecordFieldExtractor is poorly namedBeanRecordFieldExtractor is poorly namedMethods deprecated in v4.2 have been removed in v5:
JobBuilder#skip(long)
JobBuilder#limit(long)
JobBuilder#timeout(long) and JobBuilder#timeout(long, TimeUnit)
JobBuilder#silentMode(boolean)
JobBuilder#strictMode(boolean)
JobBuilder#jmxMode(boolean)
JobBuilder#call()
The keepAlive parameter has been removed since resource handling is now done by the framework ( issue #221 ). Hence the following methods have been removed from JobBuilder:
JobBuilder#reader(RecordReader recordReader, boolean keepAlive)
JobBuilder#reader(RecordReader recordReader, boolean keepAlive, RetryPolicy retryPolicy)
The "Retry Read" feature is now done with a decorator of RecordReader (issue #224 ). Hence the method JobBuilder#reader(RecordReader recordReader, RetryPolicy retryPolicy) has been removed
The RecordDispatcher API has been removed. Hence, the method JobBuilder#dispatcher(RecordDispatcher recordDispatcher) has been removed
org.easybatch.core.job.JobExecutor.execute is no more static, you need to create a JobExecutor instance to execute jobsJdbcRecordReader/JdbcRecordWriter now take a JDBC DataSource as argument instead of a Connection and the JdbcConnectionListener/JdbcTransactionListener have been removedJpaRecordReader/JpaRecordWriter now take a EntityManagerFactory as argument instead of a EntityManager and the JpaEntityManagerListener/JpaTransactionListener have been removedHibernateRecordReader/HibernateRecordWriter now take a SessionFactory as argument instead of a Session and the HibernateSessionListener/HibernateTransactionListener have been removedorg.easybatch.core.dispatcher.Predicate has moved to org.easybatch.core.writer.Predicate
org.easybatch.core.dispatcher.DefaultPredicate has moved to org.easybatch.core.writer.DefaultPredicate
org.easybatch.core.dispatcher.PoisonRecordBroadcaster has moved to org.easybatch.core.listener.PoisonRecordBroadcaster
The Batch class is no more a Record. Instead, it now contains a list of records. Hence, all interfaces and implementations related to the previous inconsistent Batch API have been removed: BatchReader, BatchFilter, BatchMapper, BatchProcessor, BatchMarshaller, BatchValidator and BatchWriter.
The "record dispatcher" concept has been removed. All dispatchers have been transformed into writers:
BroadcastRecordDispatcher -> BlockingQueueRecordWriter
ContentBasedRecordDispatcher -> ContentBasedBlockingQueueRecordWriter
ContentBasedRecordDispatcherBuilder -> ContentBasedBlockingQueueRecordWriterBuilder
RoundRobinRecordDispatcher -> RoundRobinBlockingQueueRecordWriter
RandomRecordDispatcher -> RandomBlockingQueueRecordWriter
BroadcastJmsRecordDispatcher -> BroadcastJmsQueueRecordWriter
ContentBasedJmsRecordDispatcher -> ContentBasedJmsQueueRecordWriter
ContentBasedJmsRecordDispatcherBuilder -> ContentBasedJmsQueueRecordWriterBuilder
RandomJmsRecordDispatcher -> RandomJmsQueueRecordWriter
RoundRobinJmsRecordDispatcher -> RoundRobinJmsQueueRecordWriter
Exception handling have been simplified, the following exceptions have been removed:
org.easybatch.core.reader.RecordReaderOpeningException
org.easybatch.core.reader.RecordReaderClosingException
org.easybatch.core.reader.RecordReadingException
org.easybatch.core.processor.RecordProcessingException
org.easybatch.core.dispatcher.RecordDispatchingException
org.easybatch.core.field.RecordFieldExtractionException
org.easybatch.core.mapper.RecordMappingException
org.easybatch.core.marshaller.RecordMarshallingException
org.easybatch.core.validator.RecordValidationException
org.easybatch.core.writer.RecordWritingException
ApacheCommonCsvRecord was removed. This is redundant with StringRecord.
ApacheCommonCsvRecordReader was removed. This is redundant with FlatFileRecordReader.
org.easybatch.core.processor.ComputationalRecordProcessor deprecated in v4.2 has been removedJobResult deprecated in v4.2 has been removed from JobReport, this is related to the removal of ComputationalRecordProcessor
XmlWrapperTagWriter now take a File instead of FileWriter, this is due to a resource leakBeanValidationRecordValidator is no more parametrized typePredicate is no more a parametrized typeBlockingQueueRecordReader is no more a parametrized typeBlockingQueueRecordWriter is no more a parametrized typeRecordFieldExtractor interface has been renamed to FieldExtractor
BeanRecordFieldExtractor class has been renamed to BeanFieldExtractor