A Cloud Native Unstructured Data Storage
CubeFS is a new generation cloud-native storage that supports access protocols such as S3, HDFS, and POSIX. It is widely applicable in various scenarios such as big data, AI/LLMs, container platforms, separation of storage and computing for databases and middleware, data sharing and protection，etc.
The Key Features of CubeFS
Compatible with various access protocols such as S3, POSIX, HDFS, etc., and the access between protocols can be interoperable
Support replicas and erasure coding engines, users can choose flexibly according to business scenarios
Easy to build a PB or EB-scale distributed storage service, and each module can be expanded horizontally
Supports multi-tenant management and provides fine-grained tenant isolation policies
Supports multi-level caching, multiple high-performance replication protocols, and optimizes specific performance for small files
Easy to use CubeFS in Kubernetes via CSI Driver
CubeFS has developed a CSI plugin based on the Container Storage Interface (CSI) interface specification to support cloud storage in Kubernetes clusters.
CubeFS Application Scenarios
CubeFS is a cloud-native storage infrastructure that is widely used in a variety of scenarios, including big data storage, machine learning platforms, large-scale container platforms, as well as database and middleware storage and computing separation. It is also used for data sharing and protection.
Building a Highly Reliable, Highly Available, Low-Cost, and EB-Scale Independent Key-Value Storage System with CubeFS.
As data continues to grow, businesses face greater cost challenges. In order to alleviate the storage cost pressure caused by multi-copy mode, CubeFS has introduced a erasure code subsystem (also known as BlobStore), which is a highly reliable, highly available, low-cost, and EB-scale independent key-value storage system.
CubeFS's application in storage-computing separation scenario in ClickHouse.
ClickHouse can achieve the following advantages through the data tiering set by CubeFS. Fisrt is the Ultra-large capacity, because CubeFS provides storage space with almost no capacity limit, which removes the capacity limit of ClickHouse that is restricted by local resources. Second,Higher cost-effectiveness.Compared with local SSD storage arrays, CubeFS provides a bandwidth that far exceeds SSD arrays at the same price through cluster multi-machine IO concurrency.For more information，click to get it
Big Data Cold and Hot Separation Technology Practice in OPPO
OPPO has been storing its large-scale data in HDFS clusters with a total storage capacity of EB-level. However, due to the rapidly growing business, the company faced several operational issues, including lack of machine resources, high operational costs, and high redundancy in storing cold and hot data. In order to solve these problems, OPPO implemented CubeFS
The Application of CubeFS in Machine Learning
The article discusses OPPO's machine learning platform which supports over 100 AI businesses with tens of thousands of daily training tasks. The platform uses storage to manage corpora, datasets, and business models. Due to the growth of the platform, the original storage faced challenges, so they implemented continuous architecture iterations.
Trusted by the Advanced Technology Enterprises
A young dynamic architecture firm, CUBE provides peace of mind to any developer or homeowner dedicated to excellence.
Secret of CubeFS｜ Multi-AZ Erasure Coding Disaster Tolerance
This article introduces the AZ-level disaster tolerance scheme of the Blobstore erasure coding subsystem in CubeFS. Due to the disadvantages of high storage cost and low efficiency in the multi-replica storage mode, high network bandwidth consumption in the multi-AZ case, and the need to ensure data consistency in a timely manner, we will only discuss the erasure coding (EC) storage mode. Blobstore uses erasure coding to encode and calculate user data and persistently store it in multiple AZs, achieving high availability and disaster tolerance. The main content of the article includes the EC calculation principle in multiple AZs, as well as how to maintain high availability for writing, efficient reading, and reducing cross-AZ data recovery. The reliability for EC data is not elaborated here, and can be found in our related articles.Read More
Understanding cubefs source code | The RAFT of master
This article explains the implementation details of the RAFT protocol in the Master node of the CubefS distributed file system. By analyzing the state machine of the Master node and the interaction process of the RAFT protocol, readers can better understand how the CubefS system works.Read More
Understanding cubefs source code | The HTTP server of master
This article introduces the implementation of the HTTP server in the Master node of the CubefS distributed file system. By analyzing the architecture and code implementation of the HTTP server, readers can better understand the working principle and file management function of the Master node in the CubefS system.Read More