CN116303327A - A kind of Ceph client lock optimization method and device - Google Patents

Abstract

The application provides a Ceph client lock optimization method and device. By adopting a divide-and-conquer idea, a mode that a plurality of threads respectively maintain one OSDMap object is adopted, read locking in the IO read-write process is avoided, a user-defined load balancing mapping algorithm is utilized to map loads to specific threads, and a high-efficiency multi-thread concurrency processing without locking is realized by combining a high-efficiency locking queue SPDK ring inter-thread message synchronization mechanism, so that the method can be widely applied to multi-thread locking-free solutions in high concurrency scenes.

Description

A kind of Ceph client lock optimization method and device

technical field

The present application relates to the technical fields of cloud computing and distributed storage, and in particular to a method and device for optimizing Ceph client locks.

Background technique

Ceph is a unified, distributed storage system designed for excellent performance, reliability, and scalability. It is a typical distributed system with a client-server model. During the IO read and write process of the Ceph client, a read lock will be added to the OSDMap resource object. This object contains the necessary attributes required to calculate which OSD the IO falls on according to the Crush algorithm. When the OSD failure causes the status to change, the monitor node in the cluster center will be updated. OSDMap resource object. The Ceph client needs to obtain the latest OSDMap resource object from the Monitor node to recalculate Crush and retry IO when necessary. The OSDMap resource object lock is a typical read-more-write-less lock.

The Ceph client encapsulates each IO into an Op operation, and maintains a global opsMap in the client process to record all ongoing Op operations. When there is an IO that needs to be submitted, the Op will be inserted into the Map. When the IO is completed, it will be Deleted from the Map, the Map will also be scanned periodically to handle Ops with timeout exceptions. In the original Ceph client solution, mutual exclusion locks are used to achieve synchronization between multiple threads. This method usually becomes a system performance bottleneck in a multi-threaded scenario.

Contents of the invention

In order to solve the above technical problems, the present application provides a Ceph client lock optimization method and device. This application is aimed at OSDMap resource object locks with more reads and fewer writes. By using multiple threads to maintain an OSDMap resource object, it avoids adding read locks during the IO read and write process. Under the same hardware resources, the delay is shortened and the Effectively improve the performance of the storage system. The technical scheme adopted in this application is as follows:

A kind of Ceph client lock optimization method, it is characterized in that, comprise a plurality of thread workers in the Ceph client thread pool, and specify one of a plurality of thread workers as main thread;

Each thread corresponds to an OSDMap resource object, and can access the corresponding OSDMap resource object without lock;

When updating the OSDMap resource object, the main thread is responsible for the updating process of the OSDMap resource object.

Further, the main thread is responsible for updating the OSDMap resource object, which specifically includes: when updating the OSDMap resource object, the main thread first receives a request to update the OSDMap resource object, and the local part of the main thread starts to update the corresponding OSDMap resource object , and forward the request to update the OSDMap resource object to other threads, and notify other threads to update the corresponding OSDMap resource object.

Further, after other threads have updated the corresponding OSDMap resource object, they feed back an update completion message to the main thread; if the main thread receives the update completion message fed back by all other threads, the update of the OSDMap resource object is completed, and the The main thread replies to the outside as a whole that the update of the OSDMap resource object is completed.

Furthermore, multiple thread workers complete communication through task queue and event mechanism, or use SPDK ring and SPDK polling mechanism to complete inter-thread communication.

A kind of Ceph client lock optimization method, comprises a plurality of thread workers in the Ceph client thread pool;

When each IO is encapsulated into an Op submission, the HASH algorithm is used to obtain the thread id of the Op's ownership;

Post the Op to the task queue of its own thread for processing and send an event notification, or the own thread itself will take out the Op through active polling and insert it into the opsMap of this thread for processing.

Further, after each thread receives the op_reply message replied by OSD, it uses the same HASH algorithm to obtain the belonging thread of the Op according to the tid of the message, and then delivers the task to the task queue of the belonging thread. opsMap gets the Op and processes it.

Furthermore, the entire Op processing flow does not need to add a write lock, and the sending and reply processing of the same Op are processed in the same thread.

Further, use task queue plus event notification mechanism to realize inter-thread communication, or use SPDK ring and polling mechanism to realize inter-thread communication.

A Ceph client lock optimization device, the device includes a processor and a memory storing instructions executable by the processor, and when the instructions are executed by the processor, the processor executes the above method.

A Ceph client lock optimization device, the device includes a processor and a memory storing instructions executable by the processor, and when the instructions are executed by the processor, the processor executes the above method.

Through the embodiment of the present application, the following technical effects can be obtained:

(1) In the improved solution, the HASH algorithm is used to disperse and bind Ops to business threads, and each thread only needs to maintain a part of the total data set, thus avoiding the use of mutexes, improving the system's concurrency capability, and reducing IO delay;

(2) The data set processed by each thread is relatively fixed, and the principle of cache locality can be more fully utilized to obtain better performance under limited hardware resources;

(3) For objects that read more and write less, each thread maintains a copy of the object, and the IO read and write process does not need to add a read lock, which improves the concurrency of the system.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings that need to be used in the embodiments or the description of the prior art. Obviously, the accompanying drawings in the following description are of the present application For some embodiments, those of ordinary skill in the art can also obtain other drawings based on these drawings without paying creative efforts.

Fig. 1 is a schematic diagram of an OSDMap resource object access process in the prior art;

Fig. 2 is the schematic diagram of Op processing flow in the prior art;

Fig. 3 is the schematic diagram of the OSDMap resource object update process after the improvement of the present application;

Fig. 4 is a schematic diagram of the improved Op processing flow of the present application.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of this application.

This application adopts the idea of divide and conquer for the optimization scheme of Ceph client lock. Using the custom load balancing mapping algorithm, each Op will generate an auto-incrementing unique transaction ID, and the ID will be converted into a string as the key of the FNV-1 hash function. Input, the Hash result is modulo the number of workers (number of threads) to get the worker thread that the Op belongs to. After the load is mapped to a specific thread, combined with the efficient lock-free queue SPDK ring inter-thread message synchronization mechanism, the task is forwarded to the thread processing (SPDKring is a multi-threaded efficient programming model provided by the SPDK framework, which uses a lock-free ring queue ( called ring) to achieve efficient inter-thread message forwarding). This application is a general method that can be widely used in multi-threaded lock-free solutions in high concurrency scenarios.

FIG. 1 is a schematic diagram of an OSDMap resource object access process in the prior art. The existing Ceph client process adds an OSDMap read lock process when processing IO. During normal IO read and write, each thread of the Ceph client will first obtain the OSDMap resource object read lock, query the relevant attributes of the object, and call the Crush algorithm to calculate the IO The OSD node to send to. When the OSD state changes, the OSDMap will be updated to the monitor node in the cluster center, and the OSDMap version number epoch will be incremented automatically. In some cases, the client thread will actively go to the Monitor to obtain the latest OSDMap information, and refresh the OSDMap resource object in the local memory. At this time, the update thread will first acquire the write lock, and then update the OSDMap resource object.

FIG. 2 is a schematic diagram of an Op processing flow in the prior art. The existing Op processing flow needs to add a mutex to access the opsMap. Each thread will first obtain a write lock, then insert the Op into the global ops Map, modify the relevant properties of the Op, and post it to the network thread waiting queue and notify the network thread. Release the lock. After each thread receives the op_reply message from the OSD, it first obtains the write lock, then queries the corresponding Op from the ops Map according to the transaction tid of the message, deletes the Op from the ops Map after relevant processing, and finally releases the lock.

FIG. 3 is a schematic diagram of the improved OSDMap resource object update process of the present application.

The Ceph client thread pool includes multiple thread workers, and specifies one of the multiple thread workers as the main thread;

Each thread corresponds to an OSDMap resource object, and can access the corresponding OSDMap resource object without lock;

When updating the OSDMap resource object, the main thread is responsible for the updating process of the OSDMap resource object;

The main thread is responsible for the update process of the OSDMap resource object, which specifically includes: when updating the OSDMap resource object, the main thread first receives a request for updating the OSDMap resource object, and the local part of the main thread starts to update the corresponding OSDMap resource object, and will update the OSDMap resource object. The resource object request is forwarded to other threads, and other threads are notified to update the corresponding OSDMap resource object; after other threads have updated the corresponding OSDMap resource object, they feed back the update completion message to the main thread; if the main thread receives feedback from all other threads The update completion message of the OSDMap resource object is completed, and the main thread replies to the outside world that the update of the OSDMap resource object is complete;

Multiple thread workers communicate through task queues and event mechanisms, or use SPDK ring and SPDK polling mechanisms to complete inter-thread communication.

Figure 4 is a schematic diagram of the improved Op processing flow of this application. Before multiple threads of the Ceph client concurrently initiate requests to the server, they will first obtain the OSDMap, and assemble an Op operation according to the information of the OSDMap. The Op operation will eventually Send it to the server for processing (corresponding to op_submit in Figure 4). Before the client waits for the server to complete, it will manage the Op object in the client's Ops Map. After the server completes the Op operation, it will reply the client with an ack message. When the client processes the ack message (corresponding to op_reply in Figure 4), it deletes the Op object or retries the Op operation based on the Op object.

In the improved Op processing flow, when each IO is encapsulated into an Op submission, the HASH algorithm is used to obtain the Op's belonging thread id; the Op is delivered to the task queue of its belonging thread for processing and an event notification is sent, or by The belonging thread itself takes out the Op through active polling, and inserts it into the ops Map of this thread for processing;

Subsequent processing is consistent with the original process. After receiving the op_reply message from OSD, the business thread obtains the op's belonging thread according to the tid of the message using the same HASH algorithm, and then delivers the task to the task queue of the belonging thread, and the belonging thread queries the ops Map of this thread according to the tid to obtain to Op and deal with it.

The entire Op processing flow does not need to add a write lock, and the sending and reply processing of the same Op are processed in the same thread. You can use task queue plus event notification mechanism to realize inter-thread communication, or use SPDK ring and polling mechanism to realize inter-thread communication.

To sum up, in the technical solution of this application, each thread accesses the OSDMap resource object of this thread, and when updating the OSDMap resource object, a designated main thread is responsible for the entire update process. When an OSDMap resource object is updated, the object starts to be updated locally and other threads are notified to update. After the other threads are updated, they will reply to the main thread with an update completion message.

For the ops Map lock, the improved method uses a divide-and-conquer idea. Through specific mapping rules, the Op corresponding to each IO is mapped to the back-end thread, and the original global ops Map is scattered to each thread. Each thread maintains A subset of ops Map, using task queue and event notification mechanism between threads, or SPDK ring and polling mechanism to achieve inter-thread communication. Since the data set processed by each thread is relatively fixed, the principle of cache locality can be fully utilized to improve the running speed of the program.

Although the specific embodiments of the present application have been described above, those skilled in the art should understand that these are only examples, and the protection scope of the present application is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principle and essence of the present application, but these changes and modifications all fall within the protection scope of the present application.

Claims (10)

1. The Ceph client lock optimizing method is characterized in that a Ceph client thread pool comprises a plurality of thread works, and one of the thread works is designated as a main thread;

each thread corresponds to an OSDMap resource object and can access the corresponding OSDMap resource object without lock;

when updating the OSDMap resource object, the main thread is responsible for the update process of the OSDMap resource object.

2. The method according to claim 1, wherein the main thread is responsible for an update procedure of an OSDMap resource object, in particular comprising: when updating the OSDMap resource object, the main thread receives a request for updating the OSDMap resource object, and the main thread locally starts to update the corresponding OSDMap resource object, forwards the request for updating the OSDMap resource object to other threads, and informs the other threads to update the corresponding OSDMap resource object.

3. The method of claim 2, wherein after other threads update the corresponding OSDMap resource object, an update completion message is fed back to the main thread; if the main thread receives the update completion information fed back by all other threads, the update of the OSDMap resource object is completed, and the main thread integrally replies the update completion of the OSDMap resource object to the outside.

4. A method according to any of claims 1 to 3, wherein communication is accomplished between a plurality of thread workers via a task queue plus event mechanism, or inter-thread communication is accomplished using SPDK ring and SPDK polling mechanisms.

5. A Ceph client lock optimization method is characterized in that a Ceph client thread pool comprises a plurality of thread works;

when each IO is packaged into Op for submission, the HASH algorithm is used for obtaining the Op attribution thread id;

delivering the Op to a task queue of the home thread to process and send event notification, or taking out the Op by the home thread through a polling mode, and inserting the Op into an ops Map of the thread to process.

6. The method of claim 5, wherein after each thread receives the op_reply message replied by the OSD, the home thread of the Op is obtained by using the same HASH algorithm according to the tid of the message, and then the task is delivered to a task queue of the home thread, and the home thread obtains the Op by querying the ops Map of the thread according to the tid and processes the Op.

7. The method of claim 6, wherein the entire Op process flow does not require a write lock, and the sending and replying processes of the same Op are in the same thread.

8. The method according to one of claims 5 to 7, characterized in that inter-thread communication is implemented using a task queue plus event notification mechanism or inter-thread communication is implemented using an SPDK ring and polling mechanism.

9. A Ceph client lock optimisation apparatus comprising a processor and a memory storing instructions executable by the processor, the processor performing the method of any one of claims 1 to 4 when the instructions are executed by the processor.

10. A Ceph client lock optimisation apparatus comprising a processor and a memory storing instructions executable by the processor, the processor performing the method of any one of claims 5 to 8 when the instructions are executed by the processor.

Images