What is object storage and what is the difference between file storage and block storage?

Object storage, also known as object-based storage, is a general term used to describe methods for solving and dealing with discrete units, known as objects.

Storage LAN and NAS are the two mainstream networked storage architectures, while Object-based Storage is a new networked storage architecture, and devices based on Object Storage technology are Object-based Storage The Object Storage Device Working Group of the Global Network Storage Industry Association (SNIA) released ANSI's X3T10 standard. In general, object storage combines the advantages of NAS and SANs, as well as the advantages of high-speed direct access to SANs and distributed data sharing of NAS, providing a storage architecture with high performance, high reliability, cross-platform, and secure data sharing.

Object storage is a computer data storage architecture that manages data as objects, and unlike other storage architectures, object storage, also known as object-based storage, is a general term used to describe the approach to solving and dealing with discrete units, called objects.

Object storage is often compared to valet parking at a fancy restaurant. When a customer needs to valet park, he gives the key to someone else in exchange for a receipt. The customer didn't have to know where his car was parked or how many times the waiter would move his car while he was eating.

In this metaphor, a unique identifier for a stored object represents a customer's receipt.

Characteristics of object storage:

1.Scalability, object storage can hold almost any amount of data without the need to partition the dataset.

2.Efficiency, lack of hierarchy means no bottlenecks associated with complex directory systems.

3.Availability, object storage systems have some great mechanisms to ensure data consistency, enable automatic data replication and rolling updates, and eliminate downtime.

BLU cloud object storage, there is no upper limit on storage space, and there is no need to worry about capacity expansion. It can achieve elastic scaling of storage requirements, thereby improving business flexibility. In addition, BLUE TEAM cloud object storage can be purchased on demand, which can effectively avoid idle waste of storage and bandwidth resources.

with"Exist"The corresponding other type of storage management technology is called"Virtual storage"Management Technology, abbreviated"Void"。Virtual storage management technology is a software approach to storage expansion.

The concept of virtual storage is:

The differences between object storage, file storage, and block storage are as follows: different storage devices, different characteristics, and different shortcomings.

First, the storage devices are different.

1. Object storage: The corresponding storage device of object storage is SWIFT, and the key-value storage is used.

2. File storage: The corresponding storage devices for file storage are FTP and NFS servers.

3. Block storage: The corresponding storage device of block storage is cinder, hard disk.

Second, the characteristics are different.

1. Object storage: Object storage is characterized by the high speed of block storage and the sharing of file storage.

3. Block storage: The feature of block storage is that it can be used after partitioning and formatting, which is completely different from the way of the internal hard disk of the usual host.

Third, the disadvantages are different.

1. Object storage: The disadvantage of object storage is that it is not compatible with multiple modes of parallelism.

2. File storage: The disadvantage of file storage is that the transfer rate is low.

3. Block storage: The disadvantage of block storage is that it cannot share data.

Block storage and file storage are not object storage

Object data is a structure

Unlike block storage and file storage, which manage data in the form of objects, object storage manages data in the form of objects. The biggest difference between an object and a file is that metadata is added on top of the file. In general, objects are divided into three parts:

Data, metadata, and object IDs.

The data of the object is usually unstructured data, such as: **, **, **, or document, etc.; The metadata of an object refers to the description of the object, such as: the size of the object, the owner of the document, etc.; An object ID is a globally unique identifier that is used to distinguish objects.

From the perspective of data structure, these three types of storage are fundamentally different. The data structure of block storage is array, while file storage is a binary tree (b, b-, b+, b* various trees), and object storage is basically a hash table.

Arrays and binary trees are clichés, and there is not much to say, while the hash table used by object storage is also the core data structure of the commonly heard key-value (keyvaule-type) storage, and each object finds a uid (the so-called "key" key), and the hash value (the so-called "value vaule") corresponds to the target. Find an example of a hash table as follows:

The key-value correspondence is simple and crude, after all, it is very fast to calculate a hash value, and this flat organization form can be made very large, avoiding the depth of the binary tree, and can be strongly supported for real and massive data storage and large-scale access. So not only object storage, but also many NoSQL distributed databases use it, such as Redis, MongoDB, Cassandra and Dynamo, etc.

It is a type of object storage

Object storage is divided into three storage types: standard, infrequently, and archive, which are used for frequently accessed hot data, infrequent backup data, and archived data suitable for long-term storage, covering various data storage scenarios from hot to cold.

1.Standard storage.

Standard storage provides highly reliable, high-availability, and high-performance object storage services, with high throughput and low latency service response performance, and can support frequent hot data access.

2.IA storage.

IA provides high reliability, low storage costs, and low access latency, and is suitable for long-term storage of infrequently accessed data. The unit price of IA storage is lower than that of the standard type, and the IA storage has the minimum storage time and the minimum object size, and if the storage time is less than 30 days, you will incur a certain fee if it is deleted early. If the size of a single file is less than 64KB, the storage space will be calculated based at 64KB, and data acquisition will incur charges.

Archival storage. The archive storage type provides high reliability, extremely low storage cost, and long-term storage of offline cold data, which is suitable for long-term storage (more than half a year is recommended) of archived data, which is rarely accessed during the storage cycle. With the lowest unit price among the three storage types, Archive Storage has the shortest storage time and minimum object size, and early deletion of files stored for less than 60 days incurs a fee.

If the file size is less than 64KB, the storage space will be calculated based on 64KB, and data acquisition will incur charges.

Object storage is classified into Standard, Infrequent Access Storage, Archive Storage, and Cold Archive Storage Storage.

The term "object storage" or "object-based storage" is derived from its name because it encapsulates data and metadata into an object. Metadata is essentially data that describes other data or, in the case of object-based storage, information about files that are normally stored within an object.

Objects are stored in a flat structure or address space. Each object is assigned an object ID or unique identifier, making them retrievable from a single repository or repository. Businesses value this approach a lot because it provides more flexibility to place data outside of block- and file-based storage solutions.

The differences are as follows:

1. The speed is different.

Block storage: low latency (10ms), prominent hotspots;

File storage: Different technologies vary;

Object storage: 100ms-1s, cold data;

2. The step-by-step is different.

Block storage: off-site is unrealistic;

File storage: can be distributed, but there are bottlenecks;

Object storage: high step-by-step concurrency;

3. The file size is different.

Block storage: large and small, hotspots are prominent;

File storage: suitable for large files;

Object storage: suitable for all sizes;

4. The interface is different.

Block storage: driver, kernel module;

File storage: posix;

Object Storage: RESTful API ;

5. Typical technologies are different.

Block storage: san;

File storage: hdfs, gfs;

Object storage: Swift, Amazon S3;

6. Suitable for different scenarios.

Block Storage: Banks;

File Storage: Data Center;

Object storage: network file storage;

1. Concepts and differences.

For different application scenarios, the distributed storage solutions selected will be different, so there are object storage, block storage, and file system storage. The main difference between these three is their storage interface:

1.Object Storage:

That is, the usual key-value store, and its interface is simply get, put, del and other extensions, 2Block Storage:

This kind of interface usually exists in the form of QEMU driver or kernel module, which needs to implement the interface of the block device of Linux or the block driver interface provided by QEMU, 3File Storage:

The general meaning is to support the POSIX interface, which is the same type as the traditional file system such as ext4, but the difference is that distributed storage provides parallelization capabilities, such as CEPH's CephFS (CEPHFS is Ceph's file-oriented interface), but sometimes GFS, HDFS, and other non-POSIX file-like storage interfaces are classified into this category.

2. IO features

According to these three interfaces and their application scenarios, it is easy to understand the characteristics of these three types of IO, and their correspondence in non-distributed situations is represented in parentheses:

1.Object Storage (Key-Value Database):

The interface is simple, and an object can be regarded as a file, which can only be written and read in full, usually in large files, requiring sufficient I/O bandwidth.

2.Block Storage (Hard Disk):

Its IO characteristics are the same as those of traditional hard disks, and a hard disk should be able to meet general needs, that is, it can handle the reading and writing of large files, and it can also handle the reading and writing of small files. However, hard disks are characterized by large capacity and obvious hot spots. Therefore, block storage can mainly cope with hot spots.

In addition, the latency required for block storage is the lowest.

3.File Storage (File System):

The design of the system that supports the interface of file storage is consistent with the characteristics and difficulties of traditional local file systems such as ext4, which has richer interfaces than block storage, and needs to consider the support of directories, file attributes, etc., and it should be the most difficult to implement a file storage that supports parallelization. However, it is easier for systems like HDFS and GFS to define their own standards by defining the interface according to the implementation.

Therefore, these three interfaces can be matched by the I/O characteristics of the key-value database, hard disk, and file system in a non-distributed situation. As for hot and cold, fast and slow, large and small files, it is closer to business. However, because the storage system is a generalized implementation, generally speaking, it needs to meet various requirements as much as possible, and the interface definition has cut some requirements in a certain sense, such as object storage will be more cold storage and large files.

In the era of big data, massive data brings a series of challenges to enterprises, such as storage expansion, data sharing, efficient transmission, cost and data security, when the data storage reaches the petabyte level, the shortcomings of file storage represented by NAS and block storage equipment represented by DAS and SAN in terms of scalability and operation and maintenance costs will be highlighted, which will directly lead to a large increase in equipment maintenance costs in the later stage, and at the same time, it cannot fully meet the index requirements of enterprises for the reliability, availability and security of massive storage data.

Object storage, on the other hand, is a new networked storage architecture. It avoids the shortcomings of NAS and SAN but combines the advantages of NAS and SAN, and provides a storage architecture with high reliability, cross-platform, and secure data sharing with the advantages of high-speed direct access to SANs and data sharing of NAS. In terms of the object storage currently provided by Huayun, COS public cloud object storage service and StorUltra object storage are oriented to private clouds, while StorUltra hybrid cloud data migration service is aimed at data flow, based on the massive object storage technology of distributed architecture, and has almost unlimited storage space.

At the same time, through the multi-copy data (three copies by default) and EC erasure code redundancy technology, the reliability of 10 9s and the availability of 3 9s can be easily realized, meeting the needs of users such as publishing media, medical institutions, scientific research institutes, libraries, operators, and large state-owned enterprises who have high requirements for disaster recovery and backup of important data.

You can take a look at this picture: