Parallel S3 and local filesystem execution tool.
cp
and pipe
to not omit some of the metadata flags. (#657)pipe
command. (#182)presign
command. (#634) @zemul
select
queries with more range of options to set during the query. (#494)--include
flag to cp
, rm
, and sync
commands. (#516)--show-progress
flag to cp
command. (#51)--content-disposition
flag to cp
command. (#569)--show-fullpath
flag to ls
command. (#596)--metadata
flag to cp
and pipe
commands to set arbitrary metadata for the objects. (#537)cat
command. (#245)ListObjectsV2
S3 API is enabled for Google Cloud Storage. (#617)ppc64le
support. (#552) @mgiessing
sync
command with whitespaced flag value to fail. (#541) ataberkgrl
external sort
support in sync
command which prevents sync
to an empty destination with --delete
option. (#576)sync
command, which previously caused the command to continue running even if an error was received from the destination bucket. (#564)cp
command could not upload a non-regular file to remote destination. (#618)s5cmd cp file.txt s3://bucket/a//b///c/
would copy to s3://bucket/a/b/c/file.txt
but now to s3://bucket/a//b///c/file.txt
.(#459)--endpoint-url
will not accept URLs without scheme such as example.com
. Instead, it will give an error and ask for an url with a scheme; either http://example.com
or https://example.com
(#496).--content-type
and --content-encoding
flags to cp
command. (#264)--profile
flag to allow users to specify a named profile. (#353)--credentials-file
flag to allow users to specify path for the AWS credentials file instead of using the default location.--all-versions
flag to ls
, rm
, du
and select
subcommands to apply operation on(/over) all versions of the objects. (#475)--version-id
flag to cat
, cp
/mv
, rm
, du
and select
subcommands to apply operation on(/over) a specific versions of the object. (#475)bucket-version
command to configure bucket versioning. Bucket name alone returns the bucket versioning status of the bucket. Bucket versioning can be configured with set
flag. (#475)--raw
flag to cat
and select
subcommands. It disables the wildcard operations. (#475)bench.py
script under new benchmark
folder to compare performances of two different builds of s5cmd. (#471)Disable AWS SDK logger if log level is not trace
. (#460)
Allow adjacent slashes to be used as keys when uploading to remote. (#459)
Debian packages are provided on releases page (#380)
Upgraded minimum required Go version to 1.17.
The sync command uses external sort
instead of internal
sort. This change
reduces RAM usage from ~10 GB to ~1.5 GB for sync
operation of a directory containing
1,000,000 files at a cost of speed (20% slower for 1,000,000 objects). For smaller
directories (~50,000 files) there is no significant change in speed. (#483)
Improve auto-completion support of s5cmd for zsh
and bash
, start supporting pwsh
and stop the support for fish
. Now s5cmd can complete bucket names, s3 keys in a bucket and the local files. However, install-completion
flag no longer installs the completion script to *rc
files instead it merely gives instructions to install autocompletion and provides the autocompletion script (#500).
--stat
) prints unnecessarily when used with help and version commands (#452)cp
and sync
subcommands (#360)--no-verify-ssl
flag is used. (#445)unknown url format
error when object key also includes s3://
e.g. s5cmd ls s3://foo/bar/s3://baz
(#449)endpoint-url
to have scheme (#496)signal.NotifyContext
to handle signals (#510)ci
badge URL (#546)SlowDown
as retryable (#507)notImplemented
type (#476)v2.11.2
(#495)run
command. Previously s5cmd supported a command with an inline comment like ls s3://bucket/object.gz # inline comment
. (#309)brew install peak/tap/s5cmd
. (#356)/aws
from /
. (#412)sync
command to synchronize two given buckets, prefixes, or objects. (#3)--force-glacier-transfer
flag to select
command. (#346)--ignore-glacier-warnings
flag to cp
, mv
and select
commands. (#346)--request-payer
flag to include x-amz-request-payer
in header while sending GET, POST and HEAD requests. (#297) @Kirill888
--use-list-objects-v1
flag to force using S3 ListObjects API instead of ListObjectsV2 API. (#405) @greenpau
--log=trace
) which enables SDK debug logs.(#363)run
command is not accessible. (#410)run
command. sync
was failing when it finds multiple files to remove. (#435, #436)--log
) is not valid. (#430)--stat
) is ignored when log level is error. (#359)run
command. Previously s5cmd supported a command with an inline comment like ls s3://bucket/object.gz # inline comment
. (#309)brew install peak/tap/s5cmd
. (#356)/aws
from /
. (#412)sync
command to synchronize two given buckets, prefixes, or objects. (#3)--force-glacier-transfer
flag to select
command. (#346)--ignore-glacier-warnings
flag to cp
, mv
and select
commands. (#346)--request-payer
flag to include x-amz-request-payer
in header while sending GET, POST and HEAD requests. (#297) @Kirill888
--use-list-objects-v1
flag to force using S3 ListObjects API instead of ListObjectsV2 API. (#405) @greenpau
run
command is not accessible. (#410)run
command. sync
was failing when it finds multiple files to remove. (#435, #436)run
command. Previously s5cmd supported a command with an inline comment like ls s3://bucket/object.gz # inline comment
. (#309)brew install peak/tap/s5cmd
. (#356)sync
command to synchronize two given buckets, prefixes, or objects. (#3)--force-glacier-transfer
flag to select
command. (#346)--ignore-glacier-warnings
flag to cp
, mv
and select
commands. (#346)--request-payer
flag to include x-amz-request-payer
in header while sending GET, POST and HEAD requests. (#297) @Kirill888
--use-list-objects-v1
flag to force using S3 ListObjects API instead of ListObjectsV2 API. (#405) @greenpau
run
command is not accessible. (#410)select
command. It allows to select JSON records from objects using SQL expressions. (#299) @skeggse
rb
command to remove buckets. (#303).--exclude
flag to cp
, rm
, ls
, du
and select
commands. This flag allows users to exclude objects with given pattern. (#266)--raw
flag to cp
and rm
commands. It disables the wildcard operations. It is useful when an object contains glob characters which interfers with glob expansion logic. (#235)--cache-control
and --expires
flags to cp
and mv
commands. It adds support for setting cache control and expires header to S3 objects. (#318) @tombokombo
--force-glacier-transfer
flag to cp
command. It forces a transfer request on all Glacier objects. (#206)--source-region
and destination-region
flags to cp
command. It allows overriding bucket region. (#262) @kemege
MacPorts
installation option. (#311) @manojkarthick
S3_ENDPOINT_URL
environment variable (#343) @Dexus
ed0556d Brew formula update for s5cmd version v1.3.0
e81b978 add cache-control and expires options (#318)
fb03bce all: enable -race
flag for tests (#336)
5a611da command/cp&rm: add --raw
flag (#315)
fbd99c5 command/cp: add force-glacier-transfer
flag (#312)
b4c91bd command/cp: allow overriding bucket region (#270)
d80fc02 command/expand: fix data race for concurrent writes (#330)
68f797b command/mb: fix incorrect usages in help output (#316)
d5e5ea3 command/mv: fix option usage (#338)
babc8d5 command/run: fix concurrent writes to the flags list (#335)
604f03a command: add --exclude
flag (#334)
9defce6 command: append object error to multierror to return non-zero exit code error (#328)
5495dd4 command: implement 'remove bucket' feature (#310)
c2a9d31 command: implement 'select' command (#300)
02f63b3 makefile: fix check-fmt target (#327)
89baee6 readme: add 'windows' to the list of supported operating systems (#323)
c73528f readme: add steps for installation using macports (#311)
4130a7e storage/fs: change order of precedence for URL expansion (#322)
5ba7ee0 storage/s3: do not retry on expired token (#337)