Distributed Mode

Get Software

Use the following command to build server, client, and related dependencies at the same time:

$ git clone https://github.com/cubeFS/cubefs.git
$ cd cubefs
$ make build

If the build is successful, the executable files cfs-server and cfs-client will be generated in the build/bin directory.

Cluster Deployment

Install Master

./cfs-server -c master.json

Example master.json:

Recommended

To ensure high availability of the service, at least 3 instances of the Master service should be started.

{
  "role": "master",
  "ip": "127.0.0.1",
  "listen": "17010",
  "prof":"17020",
  "id":"1",
  "peers": "1:127.0.0.1:17010,2:127.0.0.2:17010,3:127.0.0.3:17010",
  "retainLogs":"20000",
  "logDir": "/cfs/master/log",
  "logLevel":"info",
  "walDir":"/cfs/master/data/wal",
  "storeDir":"/cfs/master/data/store",
  "consulAddr": "http://consul.prometheus-cfs.local",
  "clusterName":"cubefs01",
  "metaNodeReservedMem": "1073741824"
}

Configuration parameters can be found in Master Detailed Configuration.

Install MetaNode

./cfs-server -c metanode.json

Example meta.json:

Recommended

To ensure high availability of the service, at least 3 instances of the MetaNode service should be started.

{
    "role": "metanode",
    "listen": "17210",
    "prof": "17220",
    "logLevel": "info",
    "metadataDir": "/cfs/metanode/data/meta",
    "logDir": "/cfs/metanode/log",
    "raftDir": "/cfs/metanode/data/raft",
    "raftHeartbeatPort": "17230",
    "raftReplicaPort": "17240",
    "totalMem":  "8589934592",
    "consulAddr": "http://consul.prometheus-cfs.local",
    "exporterPort": 9501,
    "masterAddr": [
        "127.0.0.1:17010",
        "127.0.0.2:17010",
        "127.0.0.3:17010"
    ]
}

For detailed configuration parameters, please refer to MetaNode Detailed Configuration.

Install DataNode

Recommended

Using a separate disk as the data directory and configuring multiple disks can achieve higher performance.

• Prepare the data directory

  • View the machine's disk information and select the disk to be used by CubeFS

  fdisk -l
  

  • Format the disk, it is recommended to format it as XFS

  mkfs.xfs -f /dev/sdx
  

  • Create a mount directory

  mkdir /data0
  

  • Mount the disk

  mount /dev/sdx /data0
  

• Start the Data Node

./cfs-server -c datanode.json

Example datanode.json:

Recommended

To ensure high availability of the service, at least 3 instances of the DataNode service should be started.

{
  "role": "datanode",
  "listen": "17310",
  "prof": "17320",
  "logDir": "/cfs/datanode/log",
  "logLevel": "info",
  "raftHeartbeat": "17330",
  "raftReplica": "17340",
  "raftDir":"/cfs/datanode/log",
  "consulAddr": "http://consul.prometheus-cfs.local",
  "exporterPort": 9502,
  "masterAddr": [
     "127.0.0.1:17010",
     "127.0.0.1:17010",
     "127.0.0.1:17010"
  ],
  "disks": [
     "/data0:10737418240",
     "/data1:10737418240"
 ]
}

For detailed configuration parameters, please refer to DataNode Detailed Configuration.

Install Object Gateway

Note

Optional section. If you need to use the object storage service, you need to deploy the object gateway (ObjectNode).

./cfs-server -c objectnode.json

Example objectnode.json, as follows:

{
    "role": "objectnode",
    "domains": [
        "object.cfs.local"
    ],
    "listen": "17410",
    "masterAddr": [
       "127.0.0.1:17010",
       "127.0.0.2:17010",
       "127.0.0.3:17010"
    ],
    "logLevel": "info",
    "logDir": "/cfs/Logs/objectnode"
}

For detailed configuration parameters, please refer to ObjectNode Detailed Configuration.

Install Erasure Coding Subsystem

Note

Optional section. If you need to use the erasure coding volume, you need to deploy it.

Please refer to Using the Erasure Coding Storage System for deployment.

Edit on GitHub

Standalone Mode Quick Use