The LeoFS Storage System
LeoFS is an Enterprise Open Source Storage, and it is a highly available, distributed, eventually consistent object/blob store. If you are seeking a storage system that can store huge amount and various kind of files such as photo, movie, log data and so on, LeoFS is suitable for that.
LeoFS is supporting the following features:
LeoFS consists of three core components - LeoStorage, LeoGateway and LeoManager which depend on Erlang.
LeoGateway handles http-request and http-response from any clients when using REST-API OR S3-API. Also, it is already built in the object-cache mechanism (memory and disk cache).
LeoStorage handles GET, PUT and DELETE objects as well as metadata. Also, it has replicator, recoverer and queueing mechanism in order to keep running a storage node and realise eventual consistency.
LeoManager always monitors LeoGateway and LeoStorage nodes. The main monitoring status are Node status and RING’s checksum in order to realise to keep high availability and keep data consistency.
You can access a LeoFS system using Amazon S3 clients and the SDK.
The presentation - Scaling and High Performance Storage System: LeoFS was given at Erlang User Conference 2014 in Stockholm on June 2014
LeoFS packages have been already provided on the Web. You're able to easily install LeoFS on your environments.
Here is the installation manual.
Here, we explain how to build LeoFS from source code. First, you have to install the following packages to build Erlang and LeoFS.
## [CentOS]
$ sudo yum install cmake check-devel gcc gcc-c++ make
## [Ubuntu]
$ sudo apt-get install gcc g++ cmake make check libtool
### For Docker
$ apt-get install lsb-release
You can install Erlang with kerl.
$ curl -O https://raw.githubusercontent.com/kerl/kerl/master/kerl
$ chmod a+x kerl
$ mkdir -p ~/bin
$ mv kerl ~/bin/
$ echo "export PATH=$PATH:~/bin" >> ~/.bashrc
$ source ~/.bashrc
$ kerl build 19.3 19.3
$ kerl list builds
19.3,19.3
$ kerl install 19.3 /path/to/19.3
$ kerl list installations
19.3 /path/to/19.3
$ source /path/to/19.3/activate
$ kerl active
The current active installation is:
/path/to/19.3
Then, clone source of LeoFS and libraries from GitHub.
$ git clone https://github.com/leo-project/leofs.git
$ cd leofs
$ git checkout -b develop remotes/origin/develop
$ ./rebar get-deps
$ ./git_checkout.sh develop
Then, build LeoFS with the following commands.
$ make && make release_for_test
Now, you can find the LeoFS package as follow.
$ ls package/
leo_gateway/ leo_manager_0/ leo_manager_1/ leo_storage/ README.md
Then, we can start and access LeoFS with the following commands. Also, you're able to easily operate LeoFS with leofs-adm script.
$ package/leo_manager_0/bin/leo_manager start
$ package/leo_manager_1/bin/leo_manager start
$ package/leo_storage/bin/leo_storage start
$ package/leo_gateway/bin/leo_gateway start
$ ./leofs-adm status
[System Confiuration]
-----------------------------------+----------
Item | Value
-----------------------------------+----------
Basic/Consistency level
-----------------------------------+----------
system version | 1.3.4
cluster Id | leofs_1
DC Id | dc_1
Total replicas | 1
number of successes of R | 1
number of successes of W | 1
number of successes of D | 1
number of rack-awareness replicas | 0
ring size | 2^128
-----------------------------------+----------
Multi DC replication settings
-----------------------------------+----------
[mdcr] max number of joinable DCs | 2
[mdcr] total replicas per a DC | 1
[mdcr] number of successes of R | 1
[mdcr] number of successes of W | 1
[mdcr] number of successes of D | 1
-----------------------------------+----------
Manager RING hash
-----------------------------------+----------
current ring-hash |
previous ring-hash |
-----------------------------------+----------
[State of Node(s)]
-------+--------------------------+--------------+----------------+----------------+----------------------------
type | node | state | current ring | prev ring | updated at
-------+--------------------------+--------------+----------------+----------------+----------------------------
S | [email protected] | attached | | | 2017-06-02 14:59:20 +0900
-------+--------------------------+--------------+----------------+----------------+----------------------------
$ ./leofs-adm start
OK
$ ./leofs-adm status
[System Confiuration]
-----------------------------------+----------
Item | Value
-----------------------------------+----------
Basic/Consistency level
-----------------------------------+----------
system version | 1.3.4
cluster Id | leofs_1
DC Id | dc_1
Total replicas | 1
number of successes of R | 1
number of successes of W | 1
number of successes of D | 1
number of rack-awareness replicas | 0
ring size | 2^128
-----------------------------------+----------
Multi DC replication settings
-----------------------------------+----------
[mdcr] max number of joinable DCs | 2
[mdcr] total replicas per a DC | 1
[mdcr] number of successes of R | 1
[mdcr] number of successes of W | 1
[mdcr] number of successes of D | 1
-----------------------------------+----------
Manager RING hash
-----------------------------------+----------
current ring-hash | 433fe365
previous ring-hash | 433fe365
-----------------------------------+----------
[State of Node(s)]
-------+--------------------------+--------------+----------------+----------------+----------------------------
type | node | state | current ring | prev ring | updated at
-------+--------------------------+--------------+----------------+----------------+----------------------------
S | [email protected] | running | 433fe365 | 433fe365 | 2017-06-02 15:00:10 +0900
G | [email protected] | running | 433fe365 | 433fe365 | 2017-06-02 15:00:12 +0900
-------+--------------------------+--------------+----------------+----------------+----------------------------
You can easily build a LeoFS cluster. See here.
About the configuration of LeoFS, See here.
You can benchmark LeoFS with Basho Bench, and here is a documentation to benchmark LeoFS.
You can test LeoFS with leofs_test whether LeoFS has issues or not before getting installed LeoFS in your dev/staging/production environment(s).
the ls command
LeoFS adheres to the versioning policy from v1.3.3.
LeoFS is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.