Multithreaded recursive directory upload to S3 using FOG
Multithreaded recursive directory uploader to S3 using fog.
It recursively transverses all contents of the directory provided as source parameter, uploading all files to the destination bucket. A destination folder where to put the uploaded files tree inside the bucket can be specified too.
By default, it uses 5 threads to upload files in parallel, but the number can be configured as well.
Files are stored as non public if not otherwise specified.
A CLI binary is included.
Add this line to your application's Gemfile:
gem 's3_uploader'
And then execute:
$ bundle
Or install it yourself as:
$ gem install s3_uploader
uploader = S3Uploader::Uploader.new({
:s3_key => YOUR_KEY,
:s3_secret => YOUR_SECRET_KEY,
:destination_dir => 'test/',
:region => 'eu-west-1',
:threads => 10
})
uploader.upload('/tmp/test', 'mybucket')
or
S3Uploader.upload('/tmp/test', 'mybucket',
{ :s3_key => YOUR_KEY,
:s3_secret => YOUR_SECRET_KEY,
:destination_dir => 'test/',
:region => 'eu-west-1',
:threads => 4,
:metadata => { 'Cache-Control' => 'max-age=315576000' }
})
Former static method upload_directory is still supported for backwards compatibility.
S3Uploader.upload_directory('/tmp/test', 'mybucket', { :destination_dir => 'test/', :threads => 4 })
If no keys are provided, it uses S3_KEY and S3_SECRET environment variables. us-east-1 is the default region.
Metadata headers are documented here
Or as a command line binary
s3uploader -r eu-west-1 -k YOUR_KEY -s YOUR_SECRET_KEY -d test/ -t 4 /tmp/test mybucket
Again, it uses S3_KEY and S3_SECRET environment variables if non provided in parameters.
s3uploader -d test/ -t 4 /tmp/test mybucket
If the :gzip
options is used, files not already compressed are packed using GZip before upload. A GZip working
directory is required in this case.
S3Uploader.upload_directory('/tmp/test', 'mybucket',
{ :s3_key => YOUR_KEY,
:s3_secret => YOUR_SECRET_KEY,
:destination_dir => 'test/',
:region => 'eu-west-1',
:gzip => true,
:gzip_working_dir => '/tmp/gzip_working_dir'
})
git checkout -b my-new-feature
)git commit -am 'Added some feature'
)git push origin my-new-feature
)Distributed under the MIT License. See LICENSE file for further details.