[go: up one dir, main page]

CN118568136A - Tendis and MongoDB-based equipment acquisition data storage management method - Google Patents

Tendis and MongoDB-based equipment acquisition data storage management method Download PDF

Info

Publication number
CN118568136A
CN118568136A CN202410663251.1A CN202410663251A CN118568136A CN 118568136 A CN118568136 A CN 118568136A CN 202410663251 A CN202410663251 A CN 202410663251A CN 118568136 A CN118568136 A CN 118568136A
Authority
CN
China
Prior art keywords
data
tendis
processed
user
mongodb
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202410663251.1A
Other languages
Chinese (zh)
Inventor
吴俊伟
张猛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Ruijing Digital Technology Co ltd
Original Assignee
Shenzhen Ruijing Digital Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Ruijing Digital Technology Co ltd filed Critical Shenzhen Ruijing Digital Technology Co ltd
Priority to CN202410663251.1A priority Critical patent/CN118568136A/en
Publication of CN118568136A publication Critical patent/CN118568136A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24552—Database cache management
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G06F12/06—Addressing a physical block of locations, e.g. base addressing, module addressing, memory dedication
    • G06F12/0646—Configuration or reconfiguration
    • G06F12/0684—Configuration or reconfiguration with feedback, e.g. presence or absence of unit detected by addressing, overflow detection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2453—Query optimisation
    • G06F16/24532—Query optimisation of parallel queries

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a device acquisition data storage management method based on Tendis and MongoDB. According to the invention, through combining the Tendis disk cache and the MongoDB device acquisition data storage management method, the high efficiency, reliability and instantaneity of device acquisition data storage and inquiry are achieved; therefore, data deduplication and caching of high concurrency data writing and query scenes are realized, direct reading and writing pressure on a database is reduced, and overall system performance is improved. In addition, the invention has good expandability and customizable performance; by adjusting the execution frequency of the timing tasks and the number of users per process, the speed and load of data processing can be flexibly controlled. Meanwhile, the data structure and the query logic adopted by the invention can meet the requirements of various different equipment data acquisition scenes.

Description

Tendis and MongoDB-based equipment acquisition data storage management method
Technical Field
The invention relates to the technical field of data storage, in particular to a Tendis and MongoDB-based device acquisition data storage management method.
Background
In recent years, with the rapid development of mobile internet and domestic and foreign personal financial credit markets, many financial credit mobile applications need to collect related data (such as device basic information, storage space, location information, application list, address book, short message record, call record, etc.) of user equipment, and upload the data to a background server for storage, so as to realize the business function of wind control approval by using device data to calculate user characteristics.
The traditional method is that the mobile client directly uploads the collected device data to a database such as MongoDB of the background for storage through an API interface. However, in a high concurrency scenario, a large amount of data directly falls into the database, which results in excessively high storage IO overhead of the database and easily causes performance bottleneck problems.
Another common approach is to use a memory cache middle layer (e.g., redis) in the background to temporarily store data and then write the cached data to the database. The scheme can relieve the direct writing pressure of the database to a certain extent, but because of limited capacity of memory caches such as Redis and the like, when the data volume is large, potential risks such as memory overflow and the like can be caused.
Therefore, the prior art has the defects, and optimization and improvement are needed.
Disclosure of Invention
The invention aims to solve the technical problems that: the device acquisition data storage management method based on Tendis and MongoDB solves the performance problems possibly occurring in high concurrency writing and query scenes.
The technical scheme of the invention is as follows: the device acquisition data storage management method based on Tendis and MongoDB comprises a data writing method and a data query method.
The data writing method comprises the following steps.
S1: and (3) data acquisition: firstly, a client collects data of various devices of a user, reports the data to a background API service through an API interface, and the background API service pushes the data to a consumption queue after receiving the data. The client is preferably a mobile client.
S2: data caching: the consumption service monitors the consumption queue data, consumes the data of the equipment to be processed, and stores the data in Tendis data cache to be processed.
S3: and (3) data storage: the data processing service obtains Tendis data to be processed through the timing task, analyzes the data and stores the data into the MongoDB.
The data query method comprises the following steps: judging Tendis whether result data are cached according to the ID of the user and the equipment type, if yes, directly reading and returning; if not, simultaneously inquiring MongoDB corresponding Collection data and Tendis user equipment data to be processed, combining and merging the data according to uniqueKey, storing the data into a Tendis result data cache, and returning. The purpose of querying the user equipment data to be processed in Tendis is to ensure that the reported user equipment data to be processed is ensured, and when the data writing scene is not completed yet, the acquired user data integrity is ensured.
Tendis is an open source, redis protocol based in-memory database that supports data persistence to disk and provides fast read and write capabilities. Tendis aims to solve some limitations of the traditional Redis in terms of large data volume, persistence and the like, and simultaneously maintains the characteristics of high performance, simplicity and easiness in use of the Redis. Tendis employ various optimization approaches such as multithreaded architecture, I/O multiplexing, etc., to improve performance and reliability.
MongoDB is a NoSQL database, which adopts a document type data storage mode and is suitable for processing a large amount of unstructured or semi-structured data. MongoDB is known for its high performance, high availability, and flexible data model. The system adopts a distributed architecture, can realize horizontal expansion, and supports automatic fault transfer and data replication so as to improve the reliability and usability of the system.
In the invention, tendis is used for caching and storing data, and by firstly storing the data into the Tendis cache, the storage pressure on the MongoDB database can be reduced, and the performance and the stability of the system can be improved. MongoDB is used as a persistent storage database and is responsible for long-term storage and management of data.
Further, in step S2, the consumption service listens to the consumption queue and consumes the data of the device to be processed, traverses each data to generate uniqueKey unique identifiers of the data characteristics of the device, processes the data, and stores the processed data in Tendis data cache of the device to be processed.
Further, the step S2 specifically includes the following steps: consuming the to-be-processed equipment data object from the consumption queue, assembling the splicing attribute field according to the equipment data type, and performing MD5 encryption to obtain uniqueKey values; then analyzing the user ID of the reported equipment data, and storing the user ID into Tendis to-be-processed equipment data user list for recording the identification of the user uploading equipment information; according to the reporting time of the reported data, calculating the sorting score, wherein the sorting score calculation rule is as follows: acquiring a second-level time stamp corresponding to the reporting time of the equipment data, and taking a negative value; if the device data reporting time is not available, taking the current time stamp and then taking a negative value; uniqueKey of the collected device data and sequencing scores thereof are added into an ordered set zset of the device data uniqueKey to be processed, and the ordered set is used for orderly storing and reading the device data according to the sequencing scores; storing the acquired device data into a hash table of the device data to be processed in the form of character strings, wherein keys are uniqueKey of the device data, and the values are JSON character strings of the device data; and removing the zset and the expiration time of the hash cache to enable the expiration time to be permanently stored, ensuring that the data which are not processed cannot be lost due to the expiration of the cache, and storing the data in the Tendis disk cache in a lasting manner.
Further, in step S3, the data processing service reads Tendis the data cache of the device to be processed at intervals by using a timing task, obtains the user device data of the specified number of configuration users, performs de-duplication filtering on the data stored in the MongoDB, and inserts the newly added data into the corresponding Collection of the MongoDB in batches, and updates the data to be updated into the corresponding Collection of the MongoDB in batches.
Further, the step S3 specifically includes the following steps: the timing task obtains a to-be-processed user ID list in a to-be-processed device data user list cache from Tendis according to the configured number userNum of each processing user; traversing each user ID of the processing user ID list, and acquiring a to-be-processed device data uniqueKey ordered set zset and all to-be-processed uniqueKey ordered sets uniqueKeySet in a cache according to the designated device type from Tendis according to the user ID; acquiring a to-be-processed device data list flows in a hash cache of the to-be-processed device data hash table from Tendis according to the specified device type according to the uniqueKey ordered set uniqueKeySet acquired in the previous step and the user ID; according to uniqueKey ordered set uniqueKeySet searched in the above steps, searching the stored device data in the corresponding Collection of MongoDB, traversing the to-be-processed device data list flows searched in the Tendis cache, storing the newly added data in a to-be-stored list, and storing the updated data in the to-be-updated list together with uniqueKey; judging a list to be saved and a list to be updated, and if the list is not empty, inserting or updating the list to the corresponding Collection of the MongoDB in batches; and deleting Tendis the processed data in the caches, wherein the data comprises three caches of a to-be-processed device data user list, to-be-processed device data uniqueKey ordered set zset and to-be-processed device data hash table hash.
Further, the interval is 10s-150s. Preferably, the interval is 50s-80s.
Further, the data query method comprises the following steps: the data using service calls a background API equipment data query interface to acquire equipment data of a user, and the background API equipment query interface judges Tendis whether corresponding equipment data exists in a result data cache according to the input user ID and equipment type, if so, the corresponding equipment data is directly read and returned to the data using service; if not, the MongoDB and Tendis are queried successively according to the user ID and the equipment type, and the uniqueKey of each piece of data in the queried equipment data set is combined, intersection data is taken and stored in a Tendis result data cache, and the intersection data is returned to the data use service.
By adopting the scheme, the invention provides a method for managing the data collection and storage of equipment based on Tendis and MongoDB, which realizes data deduplication and caching of high-concurrency data writing and query scenes, reduces the direct reading and writing pressure on a database, and further improves the overall system performance. By combining Tendis disk cache and MongoDB device acquisition data storage management method, the high efficiency, reliability and instantaneity of device acquisition data storage and inquiry are achieved. In addition, the invention has good expandability and customizable performance. By adjusting the execution frequency of the timing tasks and the number of users per process, the speed and load of data processing can be flexibly controlled. Meanwhile, the data structure and the query logic adopted by the invention can meet the requirements of various different equipment data acquisition scenes.
Compared with the prior art, the scheme of the invention has the main advantages that:
1. introducing Tendis cache based on a disk, temporarily storing high-concurrency written original data, and avoiding the possible memory overflow risk of the traditional memory cache; by utilizing the high throughput characteristic of Tendis disk cache, the original data written in high concurrency is temporarily stored, so that the performance bottleneck caused by direct writing operation on a database in a large amount is avoided;
2. Performing de-duplication processing on Tendis cached data through a timing task, and then writing the data into a MongoDB database, so that the pressure of direct database falling is reduced; the Tendis cache data are analyzed, de-duplicated and the like at fixed time, and the processed data are written into the database, so that the direct writing pressure of the database is reduced;
3. When in query, data is preferentially obtained from Tendis caches, and if the MongoDB is not hit and queried again, the query cost of the database is greatly reduced; under the data query scene, data is directly obtained from Tendis caches, so that the query cost of a database is reduced
4. And by comprehensively utilizing Tendis high throughput temporary storage capacity and MongoDB persistent storage performance, the performance of the whole system under the high concurrency data writing and inquiring scene is improved.
Drawings
FIG. 1 is a flow chart of a data writing method of the present invention;
FIG. 2 is a flow chart of the data buffering of the present invention;
FIG. 3 is a flow chart of the data store of the present invention;
fig. 4 is a flowchart of a data query method of the present invention.
Detailed Description
The invention will be described in detail below with reference to the drawings and the specific embodiments.
Referring to fig. 1, the present embodiment provides a method for managing device collection data storage based on Tendis and MongoDB, including a data writing method and a data query method.
The data writing method comprises the following steps.
S1: and (3) data acquisition: firstly, a client collects data of various devices of a user, reports the data to a background API service through an API interface, and the background API service pushes the data to a consumption queue after receiving the data. The client is preferably a mobile client.
S2: data caching: the consumption service monitors the consumption queue data, consumes the data of the equipment to be processed, and stores the data in Tendis data cache to be processed.
Specifically, the consumption service monitors the consumption queue and consumes the data of the to-be-processed device, traverses each data to generate uniqueKey unique identifiers of the data characteristics of the device, processes the data, and stores the processed data in Tendis to-be-processed device data caches. In a specific embodiment, step S2 specifically includes the steps of: : consuming the to-be-processed equipment data object from the consumption queue, assembling the splicing attribute field according to the equipment data type, and performing MD5 encryption to obtain uniqueKey values; then analyzing the user ID of the reported equipment data, and storing the user ID into Tendis to-be-processed equipment data user list for recording the identification of the user uploading equipment information; according to the reporting time of the reported data, calculating the sorting score, wherein the sorting score calculation rule is as follows: acquiring a second-level time stamp corresponding to the reporting time of the equipment data, and taking a negative value; if the device data reporting time is not available, taking the current time stamp and then taking a negative value; uniqueKey of the collected device data and sequencing scores thereof are added into an ordered set zset of the device data uniqueKey to be processed, and the ordered set is used for orderly storing and reading the device data according to the sequencing scores; storing the acquired device data into a hash table of the device data to be processed in the form of character strings, wherein keys are uniqueKey of the device data, and the values are JSON character strings of the device data; and removing the zset and the expiration time of the hash cache to enable the expiration time to be permanently stored, ensuring that the data which are not processed cannot be lost due to the expiration of the cache, and storing the data in the Tendis disk cache in a lasting manner.
S3: and (3) data storage: the data processing service obtains Tendis data to be processed through the timing task, analyzes the data and stores the data into the MongoDB.
Specifically, the data processing service reads Tendis the data cache of the device to be processed at intervals by a timing task, acquires user equipment data with the number of specified configuration users, performs de-duplication filtering on the data stored in the MongoDB, and inserts the newly added data into the corresponding Collection of the MongoDB in batches, and updates the data to be updated into the corresponding Collection of the MongoDB in batches.
In a specific embodiment, step S3 specifically includes the steps of: the timing task obtains a to-be-processed user ID list in a to-be-processed device data user list cache from Tendis according to the configured number userNum of each processing user; traversing each user ID of the processing user ID list, and acquiring a to-be-processed device data uniqueKey ordered set zset and all to-be-processed uniqueKey ordered sets uniqueKeySet in a cache according to the designated device type from Tendis according to the user ID; acquiring a to-be-processed device data list flows in a hash cache of the to-be-processed device data hash table from Tendis according to the specified device type according to the uniqueKey ordered set uniqueKeySet acquired in the previous step and the user ID; according to uniqueKey ordered set uniqueKeySet searched in the above steps, searching the stored device data in the corresponding Collection of MongoDB, traversing the to-be-processed device data list flows searched in the Tendis cache, storing the newly added data in a to-be-stored list, and storing the updated data in the to-be-updated list together with uniqueKey; judging a list to be saved and a list to be updated, and if the list is not empty, inserting or updating the list to the corresponding Collection of the MongoDB in batches; and deleting Tendis the processed data in the caches, wherein the data comprises three caches of a to-be-processed device data user list, to-be-processed device data uniqueKey ordered set zset and to-be-processed device data hash table hash.
In this embodiment, the data writing method, a specific embodiment includes the following steps.
After logging in on the client APP, the user acquires the user ID, establishes an acquisition function button, realizes the click and then invokes the SDK to acquire equipment information (such as an installed application list), and invokes a background API interface to report the assembled equipment information JSON object and the user ID to a background API service. After the back-end API interface receives the device data, after checking the validity and validity of the data, the back-end API interface pushes the data to a consumption queue (such as MQ).
As shown in fig. 2, the consumption service monitors the consumption queue, acquires the device data to be processed, analyzes the user ID, the device type and the detailed device data fields, such as an installed application list (reference fields: APP name, APP packet name, APP version number, APP version name, first installation time, latest update time, and whether or not system application), performs MD5 encryption after splicing the fields according to a set order to obtain uniqueKey values of each piece of device information, and calculates the ranking score according to the time of reporting the data. And then generating a key value according to the user ID and the device type, storing the key value into a Tendis to-be-processed device data user list, generating a key value according to the user ID, the device type and uniqueKey, storing the key value into a Tendis to-be-processed device data uniqueKey ordered set zset, generating a key value according to the user ID, the device type, uniqueKey and the detail device data, storing the key value into a Tendis to-be-processed device data hash table hash, removing the buffer expiration time, ensuring that data which is not processed cannot be lost due to buffer expiration, and storing the data in Tendis disk buffer in a lasting manner.
As shown in fig. 3, the device processing service reads Tendis to-be-processed device data caches every 10s-150s through a timing task, for example, obtains 1000 user ID lists with device types being installed applications through Tendis to-be-processed device data user list, then traverses each user ID, obtains all uniqueKey sets from Tendis to-be-processed device data uniqueKey ordered sets zset according to the user ID and the device type, further obtains all to-be-processed device data lists with single user specified device types from Tendis to-be-processed device data hash table hash, and queries a corresponding Collection of a molgodb (for example, an installed application Collection) according to the queried uniqueKey sets and user ID combination conditions to obtain data which has been stored in a durable mode. And then, according to the to-be-processed equipment data list acquired from Tendis and the equipment data traversal judgment inquired from the MongoDB, the newly added or updated data are inserted or updated into the MongoDB in batches.
The data query method comprises the following steps: judging Tendis whether result data are cached according to the ID of the user and the equipment type, if yes, directly reading and returning; if not, simultaneously inquiring MongoDB corresponding Collection data and Tendis user equipment data to be processed, combining and merging the data according to uniqueKey, storing the data into a Tendis result data cache, and returning. The purpose of querying the user equipment data to be processed in Tendis is to ensure that the reported user equipment data to be processed is ensured, and when the data writing scene is not completed yet, the acquired user data integrity is ensured.
As shown in fig. 4, in this embodiment, a specific implementation manner of the data query method is as follows: the data using service calls a background API equipment data query interface to acquire equipment data of a user (for example, an air control decision engine system acquires an installed application list calculation feature application of the user to decide air control approval), and the background API equipment query interface judges Tendis whether corresponding equipment data exists in a result data cache according to the input user ID and equipment type, if so, the corresponding equipment data is directly read and returned to the data using service; if the MongoDB and Tendis are not queried successively according to the user ID and the device type, merging according to uniqueKey of each piece of data in the queried device data set, taking intersection data, storing the intersection data into a Tendis result data cache, and returning the intersection data to the data use service.
In summary, the invention provides a method for managing data collection and storage of equipment based on Tendis and MongoDB, which realizes data deduplication and caching of high concurrency data writing and query scenes, reduces direct read-write pressure on a database, and improves overall system performance. By combining Tendis disk cache and MongoDB device acquisition data storage management method, the high efficiency, reliability and instantaneity of device acquisition data storage and inquiry are achieved. In addition, the invention has good expandability and customizable performance. By adjusting the execution frequency of the timing tasks and the number of users per process, the speed and load of data processing can be flexibly controlled. Meanwhile, the data structure and the query logic adopted by the invention can meet the requirements of various different equipment data acquisition scenes.
Compared with the prior art, the scheme of the invention has the main advantages that:
1. introducing Tendis cache based on a disk, temporarily storing high-concurrency written original data, and avoiding the possible memory overflow risk of the traditional memory cache; by utilizing the high throughput characteristic of Tendis disk cache, the original data written in high concurrency is temporarily stored, so that the performance bottleneck caused by direct writing operation on a database in a large amount is avoided;
2. Performing de-duplication processing on Tendis cached data through a timing task, and then writing the data into a MongoDB database, so that the pressure of direct database falling is reduced; the Tendis cache data are analyzed, de-duplicated and the like at fixed time, and the processed data are written into the database, so that the direct writing pressure of the database is reduced;
3. When in query, data is preferentially obtained from Tendis caches, and if the MongoDB is not hit and queried again, the query cost of the database is greatly reduced; under the data query scene, data is directly obtained from Tendis caches, so that the query cost of a database is reduced
4. And by comprehensively utilizing Tendis high throughput temporary storage capacity and MongoDB persistent storage performance, the performance of the whole system under the high concurrency data writing and inquiring scene is improved.
The foregoing description of the preferred embodiment of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, and alternatives falling within the spirit and principles of the invention.

Claims (8)

1. The equipment acquisition data storage management method based on Tendis and MongoDB is characterized by comprising a data writing method and a data query method;
the data writing method comprises the following steps:
s1: and (3) data acquisition: firstly, a client collects data of various devices of a user, reports the data to a background API service through an API interface, and the background API service pushes the data to a consumption queue after receiving the data;
S2: data caching: the consumption service monitors consumption queue data and consumes data of the equipment to be processed, and stores the data into Tendis data cache to be processed;
s3: and (3) data storage: the data processing service obtains data to be processed in Tendis through a timing task, analyzes the data and stores the data into a MongoDB;
The data query method comprises the following steps: judging Tendis whether result data are cached according to the ID of the user and the equipment type, if yes, directly reading and returning; if not, simultaneously inquiring MongoDB corresponding Collection data and Tendis user equipment data to be processed, merging and merging the data according to uniqueKey, storing the merged data into a Tendis result data cache, and returning.
2. The method for managing collected data storage of a device based on Tendis and MongoDB according to claim 1, wherein in step S2, the consumption service listens to the consumption queue and consumes the data of the device to be processed, traverses each data to generate uniqueKey unique identifiers of the data characteristics of the device, and stores the processed data in the data cache of the device to be processed Tendis.
3. The method for managing data collection and storage of a device based on Tendis and MongoDB according to claim 2, wherein step S2 specifically includes the steps of:
consuming the to-be-processed equipment data object from the consumption queue, assembling the splicing attribute field according to the equipment data type, and performing MD5 encryption to obtain uniqueKey values;
Then analyzing the user ID of the reported device data, and storing the user ID into one side of a Tendis to-be-processed device data user list for recording the identification of the user uploading device information;
According to the reporting time of the reported data, calculating the sorting score, wherein the sorting score calculation rule is as follows: acquiring a second-level time stamp corresponding to the reporting time of the equipment data, and taking a negative value; if the device data reporting time is not available, taking the current time stamp and then taking a negative value;
uniqueKey of the collected device data and sequencing scores thereof are added into an ordered set zset of the device data uniqueKey to be processed, and the ordered set is used for orderly storing and reading the device data according to the sequencing scores;
Storing the acquired device data into a hash table of the device data to be processed in the form of character strings, wherein keys are uniqueKey of the device data, and the values are JSON character strings of the device data;
And removing the zset and the expiration time of the hash cache to enable the expiration time to be permanently stored, ensuring that the data which are not processed cannot be lost due to the expiration of the cache, and storing the data in the Tendis disk cache in a lasting manner.
4. The method for managing data Collection and storage of equipment based on Tendis and MongoDB according to claim 3, wherein in step S3, the data processing service reads Tendis the data cache of the equipment to be processed at intervals by a timing task, obtains the user equipment data with the number of specified configuration users, and inserts the newly added data into the corresponding Collection of the MongoDB in batches after performing de-duplication filtering with the stored data in the MongoDB, and updates the data to be updated into the corresponding Collection of the MongoDB in batches.
5. The method for managing data collected and stored by a device based on Tendis and MongoDB according to claim 4, wherein step S3 specifically includes the steps of:
The timing task obtains a to-be-processed user ID list in a to-be-processed device data user list cache from Tendis according to the configured number userNum of each processing user;
Traversing each user ID of the processing user ID list, and acquiring all to-be-processed uniqueKey ordered sets uniqueKeySet in the to-be-processed equipment data uniqueKey ordered set zset cache according to the specified equipment type from Tendis according to the user ID;
acquiring a to-be-processed device data list flows in a hash cache of the to-be-processed device data hash table from Tendis according to the specified device type according to the uniqueKey ordered set uniqueKeySet acquired in the previous step and the user ID;
According to uniqueKey ordered set uniqueKeySet searched in the above steps, searching the stored device data in the corresponding Collection of MongoDB, traversing the to-be-processed device data list flows searched in the Tendis cache, storing the newly added data in a to-be-stored list, and storing the updated data in the to-be-updated list together with uniqueKey;
Judging a list to be saved and a list to be updated, and if the list is not empty, inserting or updating the list to the corresponding Collection of the MongoDB in batches;
And deleting Tendis the processed data in the caches, wherein the data comprises three caches of a to-be-processed device data user list, to-be-processed device data uniqueKey ordered set zset and to-be-processed device data hash table hash.
6. The method for managing data collected and stored by a device based on Tendis and MongoDB according to claim 4, wherein the interval is 10s-150s.
7. The method for managing data collected and stored by a device based on Tendis and MongoDB according to claim 4, wherein the interval is 50s-80s.
8. The method for managing data storage collected by a device based on Tendis and MongoDB according to claim 1, wherein the data query method includes the steps of: the data using service calls a background API equipment data query interface to acquire equipment data of a user, and the background API equipment query interface judges Tendis whether corresponding equipment data exists in a result data cache according to the input user ID and equipment type, if so, the corresponding equipment data is directly read and returned to the data using service; if not, the MongoDB and Tendis are queried successively according to the user ID and the equipment type, and the uniqueKey of each piece of data in the queried equipment data set is combined, intersection data is taken and stored in a Tendis result data cache, and the intersection data is returned to the data use service.
CN202410663251.1A 2024-05-27 2024-05-27 Tendis and MongoDB-based equipment acquisition data storage management method Pending CN118568136A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202410663251.1A CN118568136A (en) 2024-05-27 2024-05-27 Tendis and MongoDB-based equipment acquisition data storage management method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202410663251.1A CN118568136A (en) 2024-05-27 2024-05-27 Tendis and MongoDB-based equipment acquisition data storage management method

Publications (1)

Publication Number Publication Date
CN118568136A true CN118568136A (en) 2024-08-30

Family

ID=92470372

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202410663251.1A Pending CN118568136A (en) 2024-05-27 2024-05-27 Tendis and MongoDB-based equipment acquisition data storage management method

Country Status (1)

Country Link
CN (1) CN118568136A (en)

Similar Documents

Publication Publication Date Title
CN108319654B (en) Computing system, cold and hot data separation method and device, and computer readable storage medium
CN108268219B (en) Method and device for processing IO (input/output) request
US11625412B2 (en) Storing data items and identifying stored data items
US11567681B2 (en) Method and system for synchronizing requests related to key-value storage having different portions
KR101672901B1 (en) Cache Management System for Enhancing the Accessibility of Small Files in Distributed File System
KR20170054299A (en) Reference block aggregating into a reference set for deduplication in memory management
KR20190117001A (en) Merge Tree Modifications for Maintenance Operations
CN112100197B (en) Quasi-real-time log data analysis and statistics method based on Elasticissearch
US20140025899A1 (en) Efficiently Updating and Deleting Data in a Data Storage System
US11748357B2 (en) Method and system for searching a key-value storage
CN102819586A (en) Uniform Resource Locator (URL) classifying method and equipment based on cache
US10642530B2 (en) Global occupancy aggregator for global garbage collection scheduling
CN112241396B (en) Spark-based method and system for merging small files of Delta
CN109213760A (en) The storage of high load business and search method of non-relation data storage
CN114416742A (en) Key-Value storage engine implementation method and system
CN115576947A (en) Data management method and device, combined library, electronic equipment and storage medium
CN111831691B (en) Data reading and writing method and device, electronic equipment and storage medium
CN108415962A (en) A kind of cloud storage system
CN115934583B (en) Hierarchical caching method, device and system
CN108021562B (en) Disk storage method and device applied to distributed file system and distributed file system
CN107133334A (en) Method of data synchronization based on high bandwidth storage system
CN109947730A (en) Metadata restoration methods, device, distributed file system and readable storage medium storing program for executing
CN101853278A (en) Application method of data storage system
CN118568136A (en) Tendis and MongoDB-based equipment acquisition data storage management method
CN109800208B (en) Network traceability system and its data processing method, computer storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination