TensorFlowOnSpark Versions Save

TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.

4 years ago

initial release compatible with TensorFlow 2.x.
API changes:
- removed TFNode.start_cluster_server, which is not required for tf.keras and tf.estimator.
- removed TFNode.export_saved_model, which can be replaced by TF native APIs now.
- added TFNodeContext.num_workers to count master, chief, and worker nodes.
Spark ML Pipeline API changes:
- Scala API has been removed for now, since the Java library for TensorFlow 2.0 is not available yet.
- removed InputMode.TENSORFLOW support for ML Pipelines, since the input data is always a Spark DataFrame for this API.
- added HasMasterNode and HasGraceSecs params.
- removed optional export_fn argument for Spark ML TFEstimator (use TF export APIs instead).
- added standard default values for signature_def_key and tag_set for Spark ML TFModel.
- modified inferencing code in TFModel for TF2.x APIs.
older TF 1.x examples have been replaced with TF 2.x compatible examples.

4 years ago

last expected release compatible with TensorFlow 1.x (aside from any critical fixes), since the master branch will be moving to TF 2.0 compatibility.
handle multiple outputs with signaturedef (thanks to @markromedia).
handle exceptions after data feeding.
moved API docs to sphinx_rtd_theme.
updated to Spark 2.4.4.

5 years ago

removed tensorflow as a dependency, in order to support other variants like tensorflow-gpu or tf-nightly.
allow use of evaluator node type in cluster (thanks to @bbshetty)
refactored cluster template generation.
updated wide-deep example to use models/official code.
restore termination of feed in mnist/spark example.
updated sample notebook instructions.
updated to use Spark 2.3.3.

5 years ago

5 years ago

Added util.single_node_env(), which can be used to initialize the environment (HDFS compatibility + GPU allocation) for running a single-node instance of TensorFlow on the Spark driver.
Added an example of parallelized inferencing from a pre-trained SavedModel.

5 years ago

More deterministic GPU allocation for multi-GPU nodes.
Added timeout argument to TFCluster.shutdown() (default is 3 days). This is intended to shutdown the Spark application in the event that any of the TF nodes hang for any reason. Set to -1 to disable timeout.
Added ability to start reservation server on a specific port (contributed by @AvihayTsayeg).
Updated pipeline API for latest TF APIs (contributed by @AvihayTsayeg)
Added unit test for tf.SparseTensor support.
Updated examples to latest TF APIs (including workaround for https://github.com/tensorflow/tensorflow/issues/21745).
Updated Spark version dependency for Scala Inferencing API.
Added __version__ to module.

5 years ago

Travis CI integration for Python documentation and Scala Inferencing API builds.
Added sys.path to tensorboard search path.

5 years ago

Only set TF_CONFIG environment variable if cluster_spec has a "master", i.e. when using tf.estimator.
Updated mnist/keras/mnist_mlp_estimator.py with example of distributed/parallel inferencing via estimator.predict.
Added optional feed_timeout argument to TFCluster.train() for InputMode.SPARK.
Added optional grace_secs argument to TFCluster.shutdown().
Workaround for firewall proxy issue with get_ip_address (contributed by @viplav).
Add support for all Hadoop-compatible File System schemes (contributed by @vishnu2kmohan).
Added error messages to assert statements.
Initial Travis CI integration.

5 years ago

add grace period to TFCluster.shutdown()
add wide & deep example (contributed by @crafet)
update mnist/pipeline examples to tf.data, add instructions, and misc code cleanup (from @yileic)
parameterize versions in pom.xml and code cleanup (from @tmielika)
update Scala Inferencing pom.xml to latest tensorflow-hadoop artifact (contributed by @psuszyns)

5 years ago