A native go client for HDFS
This release contains numerous contributed bugfixes, as well a few changes in functionality:
CRC32 to CRC32C, to match the behavior of the java client. (thanks @symious)hdfs binary now supports test (https://github.com/colinmarc/hdfs/pull/292) and ls -R (https://github.com/colinmarc/hdfs/pull/289), per the posix spec. (thanks @gardenia)This release, along with several bugfixes, contains new functionality:
Client.Truncate (#73, https://github.com/colinmarc/hdfs/commit/2f114063eda00d5847ff83c07afb9ad04e90f7b4) (thanks @junjieqian!)Client.ServerDefaults (https://github.com/colinmarc/hdfs/commit/039ab59c24316d509005755f772a9a8dd7a27b5d)It contains one potentially dangerous behaviour change, in https://github.com/colinmarc/hdfs/commit/b02ab581bd500863b60a6d6718b48854c26acd23: FileWriter.Close now will correctly propagate errors in the situation where the namenode has not yet received all acks from the datanodes. Close returns a specific error in that case, ErrReplicating, which the client can either ignore or use in a retry loop. See also IsErrReplicating, for checking this case.
This release contains several contributed bugfixes, notably the longstanding HDP issues described in #260.
This overdue release contains a spate of bugfixes, as well as two major features contributed by others:
This is a small bugfix release, fixing a case where the lease renewal code didn't run, and another bug where a goroutine was possibly leaked on a call to Close. Both fixes are in 1cf907da71b60951ec9d18dfd04d10c6a66bd197.
This is a small release. In addition to some bug fixes, it includes a few useful features:
RemoveAll, and a fix such that Remove is not recursive. Please be careful to check your usage of said function, as this is a fairly major change to behavior.It also officially includes support for CDH6, although it probably worked before.
This is a major release, including multiple breaking interface changes and new features.
The library is now structured as a go module. To use it, use the import path github.com/colinmarc/hdfs/v2.
Added in #133, with lots of help from @Shastick and @staticmukesh. This adds basic kerberos authentication using to the library and the command line client - to the latter with support for ccaches. This much-requested feature should be ready for production use, but I would love your feedback and/or bug reports.
Client now has two new options, NamenodeDialFunc and DatanodeDialFunc, which can be used to replace net.Dial and set timeouts and keepalives and other useful things. You can also use SetDeadline on FileReader and FileWriter to enforce i/o timeouts. See #139 for more information.
Your hdfs.Client can be made to respect your Hadoop configuration with the hdfs.ClientOptionsFromConf method. This looks for relevant options from the configuration and tries to configure the client to match. While this doesn't do that much right now, it may be expanded to other things in the future.
The rpc package is now internal; keeping the interface backwards compatible was too difficult, and nothing in that subpackage was really useful externally anyway. rpc.NamenodeError which was possibly the only useful export, now implements an interface, hdfs.Error.
The configuration parsing/loading code and the HadoopConf type have been moved into their own package, hadoopconf. The interfaces are also slightly tweaked.
Finally, all deprecated methods, such as hdfs.NewForUser, have been removed.
This is a hotfix release, fixing an obvious bug introduced in the last release in 5160e9f9b2116d8c8cf361185427d5e919443e16.
This is a minor bugfix release, but contains one possibly breaking change.
As of #123 (change proposed by @hollow), the client will now use hostnames to connect to datanodes (when available), rather than IP addresses. If you experience problems with that, you may have an issue with your DNS configuration.
This (long-overdue) release contains a host of minor bug fixes, as well as a few new features:
hdfs put - now reads from stdin. (submitted by @Shastick)client has a new method, Walk, like filepath.Walk. (submitted by @Shastick)FileWriter now exposes Flush, for syncing data to disk.The other notable interface change is the deprecation of NewForUser and NewForConnection in 0f30457.