Kertish Dfs Versions Save

Kertish-dfs is a simple distributed storage platform, implements file storage on a single distributed computer cluster, and provides interfaces for file/folder handling. Kertish-dfs aims primarily for completely distributed operation without a single point of failure, scalable to the exabyte level.

v22.2.0147

2 years ago
  • Moved partial read responsibility to data nodes to reduce the network traffic
  • Improved resilience of data read
  • Improved head/manager node connection stability with data nodes
  • Improved consistency
  • Improved data node caching and crash prevention
  • Improved logging for problematic file chunk read request
  • Improved logging on the data node
  • Fixed opened channel for rabbitmq hook
  • Fixed deadlock when restoring the snapshot
  • Fixed faulty snapshot sync operation for data node
  • Fixed data node faulty cache usage calculation in the periodic usage report
  • Decreased maintain cycle frequency and expire duration of index cache
  • Cleaned up and refactored the code for the performance improvement

THIS VERSION IS A STABLE PRODUCTION-READY RELEASE

v21.2.0084-rc+302943

2 years ago
  • Improved error handling in dfs
  • Improved logging
  • Improved health tracking and inconsistency prevention
  • Improved index lifetime and stability across the clusters
  • Increased maintain interval from 16 hours to 24 hours
  • Added new integrity option to admin-cli structure+integrity
  • Added tree list support to head-node and filesystem tool
  • Added hook support with the extension capable for the 3rd party developers to watch dfs folder changes
  • Added HOOKS_PATH env variables to head-node
  • Added RabbitMQ hook provider to head-node hooks
  • Fixed deadlock between data-nodes and manager-node
  • Changed and separated go module usage in the project
  • Cleaned up and refactored the code for the performance improvement

THIS VERSION IS A STABLE PRODUCTION-READY RELEASE CANDIDATE VERSION

v21.2.0066

3 years ago
  • Added arm64 support for Linux and Mac OSX build
  • Redis deprecated commands are replaced
  • Tuned Redis operations to work in slow networks
  • Added REDIS_TIMEOUT env variables
  • Changed Redis can work without any password
  • Added error logging for index map dropping operations
  • Added Repair Integrity option for checksum calculation for the existing files in DFS
  • Added file checksum to be sure the integrity of the file
  • Added X-Checksum header for full file download request
  • Added new integrity option to admin-cli integrity+checksum, checksum, checksum+rebuild
  • Improved the performance of cache handling in data node
  • Added repair control to skip cluster maintain
  • Added generic error handling for filesystem tool
  • Improved error handling in administration cli
  • DEPRECATED head-address parameter in filesystem cli and added -v for version and -h for help parameters
  • DEPRECATED manager-address parameter in administration cli and added -v for version and -h for help parameters
  • Renamed CLI tools. kertish-admin became krtadm and kertish-dfs became krtfs

v20.2.0010

3 years ago
  • Improved data-node sync operation for better file handling and consistency
  • Improved integrity repair capability and coverage
  • Improved indexing consistency
  • Increased maintain interval from 8 hours to 16 hours.
  • Added periodic memory report to log for data-node
  • Upgraded package versions for "testify" and "mongo-driver"
  • Unnecessary error logging is eliminated
  • Cleaned up "!" char from logging texts
  • Fixed duplicate entries for command history on file storage shell CLI
  • Fixed crash problem related to the logger in manager-node
  • Fixed caching bug for a health check and sync processor
  • Eliminated many many bugs...

v20.2.0002

3 years ago
  • Improved cluster data nodes' connectivity handling
  • Performance Improvements for folder creation and locking
  • Replaced go-redis redis client with radix because of lightweight implementation and respectively better performance
  • NEW Snapshot feature
  • NEW Cluster balancing feature
  • NEW Cluster moving feature
  • NEW Metadata structure and data-node integrity repair feature
  • Logging is improved significantly
  • Improved indexing performance significantly
  • Improved cluster synchronization performance and data handling
  • Many stability and consistency improvements on head and data nodes
  • Fixed formatting bugs on the file-system and admin tools
  • Many refactoring and code eliminations, architectural and logical changes to improve stability and performance
  • Improved performance on all file system operations (read, delete, create, move, copy, ...)
  • Tunned failure handling and better http status code responses
  • Added Docker container capability
  • Eliminated many many bugs...

IMPORTANT NOTE The working logic of all nodes is heavily modified. The inputs and outputs are the same but how the system handling requests are changed. It is better to create a test environment to see if everything is working correctly. After upgrade, you should run consistency repair to be sure everything is aligned for the new installation.

./kertish-admin -repair-consistency

If you have any problem with the upgrade, please create an issue, I'll try to solve the incident as much as I can.

v20.1.4596

4 years ago
  • Optimized to handle frequently requested files fast and easy (data-nodes may need additional setup, please check data-node readme file)
  • Stability Improvement for Deletion
  • Zombie Notification is added to fs-tool directory list (↯)
  • Added Zombie killer option to "rm" command in filesystem tool (run kertish-fs rm -k filename.ext)
  • Descriptive lock message added at end of filenames in fs-tool directory list Ex: filename.txt (locked till 2006 Jan 02 03:04 )
  • Metadata check/fix specification is added. (run kertish-admin with -check-consistency flag)
  • Breaking Change! (for custom implementations) Header keys are changed, please check README files
  • File lock mechanism is improved
  • Limit Create and Delete sync loop to avoid infinite retry (manager-node will not try to sync data-nodes for the specific file after some times of failures. HealthCheck will handle sync at a later time.)
  • Sync improvement between data-nodes
  • Improved node prioritizing
  • Breaking Change! Changed Reservation handling. (Please update array type cluster reservations fields in MongoDB to object type) Just copy paste the following code and run on MongoDB database of Kertish-DFS.
db.cluster.find({}).forEach( function (x) {   
  var reservations = new Object();
  x.reservations.forEach(function (r) {
      Object.keys(r).forEach(function(key) {
          reservations[key] = r[key];
      });
  });
  x.reservations = reservations;
  db.cluster.save(x);
});
  • Breaking Change! Replaced Redis based mutex mechanism with Locking-Center on manager-node and head-node (Please check README files.)
  • Squeezed many bugs...

v20.1.4588

4 years ago
  • Improve node id uniqueness
  • Changed the move operation on block file
  • Changed the structure of node handshake message
  • Added port control on bind-address
  • Added root creation to head node on startup
  • Added Password support for Redis

v20.1.4575

4 years ago
  • Stabilized File System Tool
  • Added Redis Cluster Support

v20.1.4572

4 years ago
  • speed and accessibility optimization
  • file-system command-line tool summary listing is simplified
  • the project name is changed to Kertish
  • removed windows service support, only client tools
  • added file and folder joining capability
  • added fs tool shell screen for DFS FileSystem

v20.1.4569

4 years ago

First Release...