Megvii FILE Library - Working with Files in Python same as the standard library
megfile
provides a silky operation experience with different backends (currently including local file system and s3), which enable you to focus more on the logic of your own project instead of the question of "Which backend is used for this file?"
megfile
provides:
megfile
can help you easily handle it.megfile
's advantages are:
smart_open
can open resources that use various protocols. Especially, reader / writer of s3 in megfile
is implemented with multi-thread, which is faster than known competitors.smart_glob
is available on majority protocols. And it supports zsh extended pattern syntax of []
, e.g. s3://bucket/video.{mp4,avi}
.smart_exists
/ smart_stat
/ smart_sync
. If you don't find the functions you want, submit an issue.pathlib.Path
interface, referring to SmartPath
and other protocol classes like S3Path
.pip install 'megfile[hdfs]'
Path string in megfile
almost is protocol://path/to/file
, for example s3://bucketA/key
. But sftp path is a little different, format is sftp://[username[:password]@]hostname[:port]//file_path
, and relative path is replace //file_path
to /file_path
.
Here's an example of writing a file to s3 / fs, syncing to local, reading and finally deleting it.
from megfile import smart_open, smart_exists, smart_sync, smart_remove, smart_glob
# open a file in s3 bucket
with smart_open('s3://playground/megfile-test', 'w') as fp:
fp.write('megfile is not silver bullet')
# test if file in s3 bucket exist
smart_exists('s3://playground/megfile-test')
# or in local file system
smart_exists('/tmp/playground/megfile-test')
# copy files or directories
smart_sync('s3://playground/megfile-test', '/tmp/playground/megfile-test')
# remove files or directories
smart_remove('s3://playground/megfile-test')
# glob files or directories in s3 bucket
smart_glob('s3://playground/megfile-?.{mp4,avi}')
SmartPath
has a similar interface with pathlib.Path.
from megfile.smart_path import SmartPath
path = SmartPath('s3://playground/megfile-test')
if path.exists():
with path.open() as f:
result = f.read(7)
assert result == b'megfile'
$ megfile --help # see what you can do
$ megfile ls s3://playground/
$ megfile ls -l -h s3://playground/
$ megfile cat s3://playground/megfile-test
$ megfile cp s3://playground/megfile-test /tmp/playground/megfile-test
pip3 install megfile
You can specify megfile version as well
pip3 install "megfile~=0.0"
megfile can be installed from source
git clone [email protected]:megvii-research/megfile.git
cd megfile
pip3 install -U .
git clone [email protected]:megvii-research/megfile.git
cd megfile
pip3 install -r requirements.txt -r requirements-dev.txt
Using s3
as an example, the following describes the configuration methods. For more details, please refer to Configuration.
You can use environments and configuration file for configuration, and priority is that environment variables take precedence over configuration file.
You can use environments to setup authentication credentials for your s3
account:
AWS_ACCESS_KEY_ID
: access keyAWS_SECRET_ACCESS_KEY
: secret keyOSS_ENDPOINT
/ AWS_ENDPOINT_URL_S3
/ AWS_ENDPOINT_URL
: endpoint url of s3AWS_S3_ADDRESSING_STYLE
: addressing styleYou can update config file with megfile
command easyly:
megfile config s3 [OPTIONS] AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY
$ megfile config s3 accesskey secretkey
# for aliyun
$ megfile config s3 accesskey secretkey \
--addressing-style virtual \
--endpoint-url http://oss-cn-hangzhou.aliyuncs.com \
You can get the configuration from ~/.aws/credentials
, like:
[default]
aws_secret_access_key = accesskey
aws_access_key_id = secretkey
s3 =
addressing_style = virtual
endpoint_url = http://oss-cn-hangzhou.aliyuncs.com
We welcome everyone to contribute code to the megfile
project, but the contributed code needs to meet the following conditions as much as possible:
You can submit code even if the code doesn't meet conditions. The project members will evaluate and assist you in making code changes
Code format: Your code needs to pass code format check. megfile
uses yapf
as lint tool and the version is locked at 0.27.0. The version lock may be removed in the future
Static check: Your code needs complete type hint. megfile
uses pytype
as static check tool. If pytype
failed in static check, use # pytype: disable=XXX
to disable the error and please tell us why you disable it.
Note : Because pytype
doesn't support variable type annation, the variable type hint format introduced by py36 cannot be used.
i.e.
variable: int
is invalid, replace it withvariable # type: int
Test: Your code needs complete unit test coverage. megfile
uses pyfakefs
and moto
as local file system and s3 virtual environment in unit tests. The newly added code should have a complete unit test to ensure the correctness
You can help to improve megfile
in many ways:
megfile
repo.megfile
to your friends.