CN116126995A - Index information generation method and device and computer readable storage medium - Google Patents

Abstract

Embodiments of this specification provide a method, device, and computer-readable storage medium for generating index information, which includes: after each machine learning model is trained, acquiring the first type of data and the second type of data generated by the training; The first type of data of each machine learning model is saved to the model file data set of the model training platform, and the second type of data is saved to the text file data set of the model training platform; according to the model file data set and the text file The data set generates file index information; the file index information includes access entry information of the first type of data and the second type of data corresponding to the machine learning model. This method uses the model training platform to train multiple models. During the training process, the data generated by the training is continuously obtained and stored in a unified manner, which solves the technical problem that team members cannot conveniently view the experimental results of other members of the team. Improved efficiency for team members viewing test results.

Description

Method, device and computer-readable storage medium for generating index information

technical field

The present disclosure relates to the technical field of machine learning, and in particular to a method, device and computer-readable storage medium for generating index information.

Background technique

Machine learning and big data are becoming more popular in recent years, and their impact on society is expanding. Many industries increasingly rely on machine learning algorithms and artificial intelligence models to make critical decisions that affect businesses and individuals every day. In a complete machine learning model experiment life cycle, there are many machine learning artifacts, such as datasets, model training code, model experiment index evaluation data, model files, etc.

In the existing technology, for a team-based multi-machine learning model development task, the team members cannot conveniently view the experimental results of other members of the team.

Contents of the invention

In view of this, multiple implementations of this specification are dedicated to providing a method for generating index information to solve to a certain extent in the development task of a team-based multi-machine learning model, where team members cannot conveniently view the technical issues with the experimental results of other members of the

Multiple implementations in this specification provide a method for generating index information, the method is applied to a model training platform, and the method includes: acquiring the first type of data and the second type of data generated by machine learning model training; wherein, the first One type of data includes at least model file data of the machine learning model, and the second type of data includes at least code file data of the machine learning model and metadata of the model file. The first type of data of each machine learning model is saved to the model file data set of the model training platform, and the second type of data is saved to the text file data set of the model training platform. File index information is generated according to the model file data set and the text file data set; wherein, the file index information includes access entry information of the first type of data and the second type of data of the machine learning model.

An embodiment of the present specification provides a method for displaying index information, the method comprising: receiving index information sent by a model training platform; wherein, the index information is obtained by using the aforementioned method for generating index information. An index page is formed according to the index information; the index page includes access entry information corresponding to the first type of data and the second type of data of the machine learning model or a text identifier bound to the access entry information.

One embodiment of this specification provides a device for generating index information, the device is applied to a model training platform, and the model training platform is used for the training of multiple machine learning models; the device includes: an acquisition unit for Obtain the first type of data and the second type of data generated by machine learning model training; wherein, the first type of data includes at least the model file data of the machine learning model, and the second type of data includes at least the code file data and model of the machine learning model Metafile data for the file. The storage unit is used to store the first type of data of each machine learning model into the model file data set of the model training platform, and store the second type of data into the text file data set of the model training platform. A generating unit, configured to generate file index information according to the model file data set and the text file data set; wherein, the file index information includes access to the first type of data and the second type of data corresponding to the machine learning model Entry information.

An embodiment of the present specification provides an electronic device, including a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program.

One embodiment of the present specification provides a computer-readable storage medium, where a computer program is stored in the readable storage medium, and the above-mentioned method is implemented when the computer program is executed by a processor.

In the multiple implementations provided in this specification, in a team-based multi-machine learning model development task, multiple machine learning models are trained by using the model training platform, and the corresponding first type of data is obtained after the training is completed And the second type of data, and then store them in a unified manner to form a model file dataset and a text file dataset, and then build the index information of the file, so that team members can easily view the experimental results of other members through the index information, realizing the realization of team members. It can quickly view the experimental results, which improves the efficiency of viewing and comparison.

Description of drawings

FIG. 1 is a schematic diagram of an application environment of a method for generating index information provided by an embodiment of this specification.

FIG. 2 is a schematic flow chart of a method for generating index information provided by an embodiment of this specification.

Fig. 3 is a block diagram of a model training platform provided by an embodiment of this specification.

FIG. 4 is a logical schematic diagram of a working process of a model training platform provided by an embodiment of this specification.

FIG. 5 is a schematic flowchart of a working process of a model training platform provided by an embodiment of the present specification.

FIG. 6 is a schematic flowchart of a method for displaying index information provided by an embodiment of the present specification.

Fig. 7 is a block diagram of an index information generation device provided by an embodiment of this specification.

Fig. 8 is a block diagram of an electronic device provided by an embodiment of this specification.

Detailed ways

In related technologies, machine learning uses some algorithms to analyze data and learn from it to obtain a machine learning model. The obtained machine learning model can be used to reason and predict some data, so as to achieve some specific tasks, such as classification tasks. The machine learning model may be a neural network model, a linear network model, a deep learning model, a support vector machine or other types of machine learning models obtained after training. The model is a function that uses some algorithms and learns from data to achieve a specific mapping. This machine learning model file is capable of recognizing certain types of patterns. The machine learning model file generally includes a file describing the model structure (such as the structure of a convolutional neural network), such as a meta file; it also includes a file describing model parameters (such as connection weight parameters between layers), such as a ckpt file.

In a complete machine learning task, various types of data are usually involved, such as data for machine learning model training, code file data for model training, model evaluation index file data, and model file data. Therefore, it is very important to track and manage each data in a task. For a team-based multi-machine learning model development task, in the existing scenario, some team members may develop locally. After the training is completed, the model file data and model evaluation index file data are directly stored locally or stored on a file server. Some team members may also develop in the model development server. After the training is completed, the model file data and model evaluation index file data are stored in the cloud, and some team members even do not manage the above data. Therefore, it is inconvenient for members of the entire team to view each other's experimental results, and a member of the team cannot easily view the experimental results of other members of the team. It is also not convenient to share experimental results among team members.

To sum up, it is necessary to provide a method for generating index information, by using a model training platform to train multiple machine learning models, and after the training is completed, obtain the corresponding first-type data and second-type data, and then Unified storage solves the technical problem that team members cannot easily view the experimental results of other members of the team, realizes the rapid viewing of experimental results by team members, and improves the efficiency of viewing and comparison.

As shown in FIG. 1 , the embodiment of this specification provides a system for generating index information, and the system for generating index information may include a terminal and a server. The server may be an electronic device with certain computing and processing capabilities. For example, the server may be a server of a distributed system, or a system having multiple processors, memories, network communication modules, etc. that operate cooperatively. The server can also be a cloud server, or an intelligent cloud computing server or an intelligent cloud host with artificial intelligence technology. The server may also be a server cluster formed for several servers. Or, with the development of science and technology, the server may also be a new technical means capable of realizing the corresponding functions of the embodiments described in the description. For example, it can be a new form of "server" based on quantum computing.

In the implementation manner of this specification, the terminal may be an electronic device with network access capability. Specifically, for example, the terminal may be a desktop computer, a tablet computer, a notebook computer, a smart phone, a digital assistant, a shopping guide terminal, a TV, and the like. Alternatively, the terminal may also be software that can run on the electronic device.

The network can be any type of network that can support data communications using any of a number of available protocols, including but not limited to TCP/IP, SNA, IPX, and the like. The network or networks can be a Local Area Network (LAN), Ethernet-based network, Token Ring, Wide Area Network (WAN), Internet, Virtual Network, Virtual Private Network (VPN), Intranet, Extranet, Public Switched Telephone Network (PSTN) PSTN), infrared networks, wireless networks (eg Bluetooth, WIFI) and/or any combination of these and/or other networks.

The system for generating index information may also include one or more databases. For example, a database used by a server may be local to the server, or may be remote from the server and may communicate with the server via a network-based or dedicated connection. Databases can be of different types. In some embodiments, the database used by the server may be a relational database. One or more of these databases may store, update and retrieve the database and data from the database in response to commands.

As shown in FIG. 2 , an embodiment of this specification provides a method for generating index information. The method for generating index information is applied to a model training platform, and the model training platform is used for training multiple machine learning models. The method for generating the index information may include the following steps.

Step S101: After each machine learning model is trained, obtain the first type of data and the second type of data generated by the training; wherein, the first type of data includes at least the model file data of the machine learning model, and the second type of data includes at least Including the code file data of the machine learning model and the metadata file data of the model file.

In some cases, for a team-based multi-machine learning model development task, it may be necessary to develop multiple different types of models, and there may also be different versions of the same type of model. Many types of data and corresponding files are involved in the development process of a specific model. Specifically, training data for training needs to be prepared before training is performed. The training data may be in the form of a data set file, and different types of preprocessing need to be performed on the data set file to form different versions of the data set file.

After multiple iterations, the loss function converges, and the model file data of the machine learning model and the evaluation index file data of the machine learning model are obtained. After analyzing the data of the machine learning model evaluation index file, it is found that the corresponding evaluation index of the machine learning model obtained this time is very different from the expected evaluation index, so it may need to be retrained one or more times. The hyperparameters of the machine learning model, the network parameters of the model, and the structure of the model may need to be modified before retraining. Therefore, multiple different versions of model code file data will be obtained. After retraining, the model file data and machine learning model evaluation index file data of the corresponding machine learning model will also be obtained.

From the above process, it can be seen that in the life process of a complete machine learning model training, various types of data will be involved. These data need to be managed, and the same type of data will also have different versions. Therefore, a unified data version management method is needed.

In this embodiment, the above-mentioned data may be classified into the first type of data and the second type of data according to the size of the data. The first type of data may include model file data of machine learning models. In some embodiments, the model file data of the machine learning models obtained by training under different machine learning frameworks are different. For example, the model file data of the machine learning model obtained by training with the keras framework mainly includes file data in h5 format. The file data in the h5 format includes the structure of the model, the weight of the model, the training configuration and the state of the optimizer. The model file data of the machine learning model obtained by training with the Tensorflow framework mainly includes file data in meta format and file data in ckpt format. The file data in the meta format is mainly used to save the model diagram structure. The file data in the ckpt format is mainly used to save variables such as network weight parameters.

In this embodiment, in addition to the model file data of the machine learning model, the first type of data may also include the machine learning model evaluation indicator file data and machine learning model training data. Wherein, the data for machine learning model training may include image data, video data, text data, etc. used as training samples. The data of the evaluation index file of the machine learning model is the data representing the performance index of evaluating the machine learning model. The performance indicators can include accuracy (Accuracy), precision (Precision), recall (Recall), P-R curve (Precision-Recall Curve), F1 Score and so on.

The second type of data may include code file data of the machine learning model and metadata file data of the model file. The code file data represents codes that are specifically run in the machine learning framework. Metafile data may be used as index data of the specific storage location of the first type of data, and the metadata may include the specific storage location of the first type of data and the data volume of the first type of data. In some implementations, a cryptographic hash function may be used to calculate a hash value of the first type of data, store the hash value into the metadata, and modify the file name of the first type of data accordingly. In this way, the uniqueness of the MD5 value can be used to distinguish multiple first-class files. Furthermore, through the specific storage location, data volume and hash value corresponding to the first-class data in the metadata, it can be easily found The first type of data required.

In this embodiment, some tracking tools of the model training process can be used to obtain the first type of data and the second type of data, for example, the MLFLOW tool can be used to track the model training process. In order to make the MLFLOW tool applicable to the model training platform, secondary development of MLFLOW can be carried out. By combining the Http protocol, the MLFLOW tool can be called through an access request, so that the MLFLOW tool can be applied to the machine learning model deployed in the tracking model training platform.

In a team-based multi-machine learning model development task, the MLFLOW tool can be used to obtain the first type of data and the second type of data in a timely manner. In some implementations, other tracking tools for the model training process can also be used. For example, TensorBoard, Trains, etc.

In this embodiment, as shown in FIG. 3 , a development team can train multiple machine learning models by using a model training platform, and the model training platform is mainly for realizing an online programming environment. The model training platform can be configured on the server side or in a cloud environment. The model training platform may include a model training module, a data version control module, and a storage module. Specifically, the model training module can be implemented through cloud technology and container technology, and all team members can perform model training in the model training module. And the MLFLOW tool is also configured to track the experimental results. The data version control module includes a DVC (data version control, data version management) module and a GIT (an open source distributed version control system) module. The data version control module is used for version control of experimental results. The storage module is used to uniformly store the experimental results. Specifically, the data version control module can carry out secondary development for the DVC module and the GIT module, package it into a software service, and provide an access port based on the Http protocol. When the model training platform needs to call the DVC module or GIT module, it can send an access request to the access port through the Http protocol, and can cause the DVC module or GIT modules perform corresponding functions.

In this embodiment, please refer to FIG. 4 , the labeler can perform sample labeling on the model training platform to obtain a sample data set. Algorithm engineers can use the online programming environment provided by the model training platform for model development and model debugging. Furthermore, the machine learning model after model debugging can be used for model training using the sample data set, and a training result can be obtained. The training results include the model file data of the machine learning model, the index file data and hyperparameter file data of the machine learning model, and the metadata file data corresponding to the generated model file data of the machine learning model and the machine learning model evaluation index file data. Algorithm engineers can verify whether the effect of the trained machine learning model is ideal based on the model training platform. In the case that it is not ideal, you can manually adjust the parameters of the model file data, and after model debugging, perform the model training process again. Or, if it is not considered ideal, you can also feed back some data optimization suggestions for the sample data to optimize the sample data in the sample data set. If you think that the machine learning model has achieved the desired effect, you can end the processing of the machine learning model.

In some implementation manners, the above-mentioned tracking tool for the model training process may be pre-configured in the online programming environment. For example, the MLFLOW tool. Machine learning frameworks can also be pre-configured. For example, machine learning frameworks such as Pytorch, TensorFlow, XGBoost, scikit-learn, etc. In some implementations, the above-mentioned online programming environment may be implemented in the following manner. For example, when the model training platform is configured in a cloud environment, members of the development team can input configuration information through an application program interface (application program interface, API) or a graphical user interface (graphical user interface, GUI). The configuration information is sent to the model training platform. The model training platform creates a corresponding Docker based on the configuration information. When performing training, start Docker, and mount GPU and storage to Docker according to the configuration information. Docker executes the machine learning framework and MLFLOW tool according to the configuration information, and reads the dataset file for training. In some embodiments, if a member of the development team needs to modify the code of the machine learning model, Jupyter NoteBook can be used to adjust the code of the machine learning model online.

In this embodiment, considering the characteristics of the machine learning model development task, the files obtained after the training of each machine learning model are divided into the first type of data and the second type of data. It lays the foundation for subsequent file storage management.

Step S102: Save the first type of data of each machine learning model to the model file data set of the model training platform, and save the second type of data to the text file data set of the model training platform.

In some cases, after obtaining the first type of data and the second type of data through the tracking tool of the model training process, these files need to be stored for subsequent viewing or use.

In this embodiment, the model file data set is used to store the first type of data. Correspondingly, the text file dataset is used to store the second type of data. In some implementations, the model file dataset may employ a remote repository. For example, FS/HDFS/NFS/NAS, etc. Text file datasets can use remote code repositories. For example, Gitee/Gitlab/Github etc. Specifically, for the above-mentioned different data types (model file data of machine learning models, file data of machine learning model evaluation indicators, data for machine learning model training, code file data of machine learning models, and corresponding metadata file data), a A common data storage solution for unified data version management. Wherein, for the model file data of the machine learning model, the machine learning model evaluation index file data and the machine learning model training data, it is first input into a password hash function to generate a hash value, and the corresponding hash value is stored In the corresponding metadata file data, name the file name of the first type of data with the hash value corresponding to the file, and store it in the remote storage repository. Store the corresponding metadata file data in the remote code repository. In this way, an association relationship can be established between the metadata file data and the first type of data through the hash value. The first type of data can be found conveniently through the content of the metadata file.

Code data for the machine learning model is stored in a remote code warehouse. For the code file data of the machine learning model, Git can be used for code version management. For the above-mentioned model file data of the machine learning model, machine learning model evaluation index file data, and machine learning model training data, Git is also used for file version control to reduce dependence on other components. But for binary files, Git needs to store the changes for each submission, and every time the binary file modification changes, additional submissions will be generated. This will lead to a large increase in the amount of data, and the volume of the remote code warehouse will also increase rapidly. In order to reduce the size of the remote code warehouse itself, in some implementations, DVC can be introduced to perform data version control. For example, only the model file data of the machine learning model, the machine learning model evaluation index file data, and the metadata file data of the machine learning model training data are stored in the remote code repository. The model file data of the machine learning model, the machine learning model evaluation index file data, and the machine learning model training data are stored in the remote repository.

In this embodiment, the action of saving can be realized by the two tools of Git and DVC. Specifically, the code file data of the machine learning model, the model file data of the machine learning model, the evaluation indicator file data of the machine learning model, and the metadata of the machine learning model training data are processed through Git. The model file data of the machine learning model, the machine learning model evaluation index file data and the data for training the machine learning model are processed through the DVC. More specifically, the real data source (the model file data of the machine learning model, the machine learning model evaluation indicator file data, and the machine learning model training data) is first transmitted to the remote repository through the DVC. Then use git push to push the mapping data of the DVC to a remote code repository, such as github or gitlab, where the mapping data of the DVC is also the above-mentioned metadata file data.

In some implementations, the directory structure of the machine learning model's model file data, machine learning model evaluation indicator file data, machine learning model training data and code file data in a specific development task of a certain machine learning model can be specified . Through a unified directory structure, to ensure that team members can easily locate related problems when troubleshooting. At the same time, adopting a unified directory structure can realize the multi-version data merging function, and by defining corresponding specification data, the conflict of data files can be effectively avoided. The unified directory structure can include the following six first-level directories. Specifically, dvc directory (data version management directory), git directory (code version management directory), data_set directory (data directory for machine learning model training), model_file directory (model file data directory for machine learning models), model_metric directory (machine Learning model evaluation index file data directory), source_code directory (code file data directory).

In this implementation manner, a specific example may be used to illustrate how to perform the storage operation on the first type of data and the second type of data. See Figure 5. Specifically, gitlab is used to store text file datasets, hdfs is used to store model file datasets, scikit-learn is used as the machine learning framework, and MLflow is used as the tracking tool for the model training process, for example. First, you can create an experimental task for a new local machine learning model in the model training platform, and judge whether the output directory of the new machine learning model exists. If it exists, do nothing. If it does not exist, create a new directory. Release branch, and switch to the new branch. Algorithm engineers can use the online editing environment provided by the model training platform to develop and debug the model, specifically, formulate the initial version of the machine learning model based on the machine learning framework, and debug the machine learning model.

Next, you can use the data set formed by the sample data marked by the labeler on the model training platform to carry out model training for the machine learning model. MLflow tracks the training process and monitors whether the training is successful. If the training fails, then end the experimental task; if the training is successful, end the experimental task after obtaining the training results, the training results include the model file data of the machine learning model and the machine learning model evaluation index file, and generate the model file data of the machine learning model Metafile data corresponding to the machine learning model evaluation indicator file data. Add metadata to the staging area. Submit the metafile data to the local code warehouse, and then push the metafile data in the local code warehouse to gitlab. The DVC can be called to transfer the model file data of the machine learning model and the machine learning model evaluation indicator file data to hdfs for storage.

Step S103: Generate file index information according to the model file data set and the text file data set; wherein, the file index information includes the access entry information of the first type of data and the second type of data of the machine learning model .

In some cases, for a team-based multi-machine learning model development task, after realizing the tracking of the development process and the unified storage and version control of the data, it is also necessary for team members to easily view the Experimental results of other members.

In this implementation manner, the file index information may include access entry information of the first type of data and the second type of data. The access entry information is represented as information corresponding to the actual storage address of the first type of data and the second type of data of the machine learning model. In some implementations, in addition to the access entry information of the first type of data and the second type of data corresponding to the machine learning model, the file index information may also include the first type of data and the second type of data corresponding to the machine learning model Information carried by the data itself. Specifically, for example, the information carried by the code file data may include specific code content data and corresponding versions. The information carried in the machine learning model evaluation index file data may include the evaluation index corresponding to a certain machine learning model. Team members can query the information included in the first type of data and the second type of data according to the file index information. The first type of data and the second type of data may also be downloaded to the local according to the file index information.

Specifically, in some implementations, the first type of data may include model file data and machine learning model evaluation index file data of the machine learning model. The second type of data may include code file data of the machine learning model, model file data, and metadata file data of the evaluation index file data. In this case, team members can query the model file data of a certain machine learning model according to the file index information, so as to obtain the model parameters, network architecture, etc. of the machine learning model. Team members can query the machine learning model evaluation index file data according to the file index information, so as to obtain the specific evaluation index of the machine learning model. Team members can query the metadata of the machine learning model evaluation index file data according to the file index information, and then determine the actual storage of the machine learning model evaluation index file data according to the metadata of the machine learning model evaluation index file data location, so as to download the machine learning model evaluation index file data to the local. Furthermore, team members can also choose whether to retrain the machine learning model according to the specific evaluation index.

In this embodiment, file index information is generated according to the model file data set and the text file data set. Team members can easily view the experimental results of other members of the team based on the file index information. It solves the technical problem that team members cannot conveniently view the experimental results of other members of the team, realizes the rapid viewing of experimental results by team members, and improves the efficiency of viewing and comparison. In this embodiment, a model file data set and a text file data set are provided to store the first type of data and the second type of data. Therefore, it is convenient to perform disaster recovery processing on the first type of data and the second type of data, thereby avoiding data loss caused by disk damage. Correspondingly, this centralized management method of the first type of data and the second type of data is also convenient for version management of the experimental results, and convenient for tracking the experiments.

In some implementations, before the step of obtaining the first type of data and the second type of data generated by the training after each machine learning model is trained, it also includes: receiving a data port to perform multiple machine learning A request for a model training task; wherein, the request includes configuration information of the machine learning model training task; according to the configuration information, a corresponding workspace is created for the plurality of machine learning model training tasks; in the workspace Execute the machine learning model training task.

In some cases, for a team-based multi-machine learning model development task, at a certain time node, only one team program needs to execute the machine learning model training task, and it needs to execute multiple machine learning model training tasks at the same time. Condition. Therefore, for the situation where there is only one team member but multiple machine learning models are developed, unified management and storage of files is also required.

The one data port means that a certain team member can log in to the model training platform through the account of the terminal device to perform machine learning model training tasks based on a terminal device. The team member may log in to a web page configured on the terminal, or may be a client configured on the terminal, or another type of user terminal, and the like. The team member submits requests for performing multiple machine learning model training tasks on the web page or client. The request is then sent to the server.

The configuration information may be a specific configuration file. Specifically, when a certain team member inputs corresponding information on the web page or client, the web page or client generates a configuration file according to the information. Wherein, the information may be the type of model training framework, code version, operating parameters, and so on. Or, the team members directly select the corresponding configuration through the configuration template provided by the web page or the client, and then the web page or the client generates the configuration file. The configuration file is carried in the request sent by the account to execute multiple machine learning model training tasks. After receiving the configuration file, the server parses the configuration file to obtain the corresponding multiple machine learning model training tasks. The type of model training framework, code version, running parameters, path of the dataset file, etc. Wherein, the operating parameters may also include system version, memory size, number of GPU cards, and the like.

The workspace described above can be expressed as using containerization technology to build an isolated and complete training environment through a mirror image. Construct corresponding workspaces or training environments for multiple machine learning model training tasks. Specifically, in some implementations, a container instance can be quickly deployed to run an interactive development workbook (such as Jupyter Notebook), a tracking tool for the model training process (such as MLflow), and data Version control tools (Git and DVC). A certain team member A can log in to the interactive development workbook to control the training process of multiple machine learning models. Scale, you can modify the hyperparameters of a machine learning model, etc. At the same time, the experimental results of the multiple machine learning models can be collected by the tracking tool of the model training process, and then stored separately through the data version control tool. At this point, another team member B can log in to the web page or client to view the training results of the multiple machine learning models. For example, specific training results can be rendered to the front-end page by MLflow for display.

This method is aimed at the situation where there is only one team member but multiple machine learning model training tasks need to be performed. Use containerization technology to create a corresponding workspace for each machine learning model training task, and execute the corresponding machine learning model training task in this workspace. Correspondingly, other team members who do not need to perform machine learning model training tasks can also log in Check the experimental results of the multiple machine learning models in the model training platform. This method takes into account a special situation in the development task of team-based multi-machine learning models, ensures that the team members can quickly view and compare experimental results, and improves the efficiency of team-based development tasks.

In some implementations, before the step of acquiring the first type of data and the second type of data generated by machine learning model training, it also includes: receiving requests from multiple data ports to perform machine learning model training tasks; wherein, the The request includes executing multiple machine learning model training tasks and configuration information of the machine learning model training tasks; creating a corresponding workspace for the data port according to the configuration information; executing the machine learning in the workspace Model training tasks.

In some cases, for a team-based multi-machine learning model development task, at a certain time node, there are multiple team members who need to perform machine learning model training tasks. And where at least one team member needs to perform multiple machine learning model training tasks at the same time. Therefore, unified management and storage of files is also required for this situation.

The multiple data ports indicate that multiple team members have logged into the model training platform through their respective terminal devices to perform machine learning model training tasks. The creating a corresponding workspace for the account according to the configuration information means creating a corresponding workspace for each data port. That is, one data port corresponds to one container instance. Among them, some container examples need to perform one machine learning model training task, and some container examples need to perform multiple machine learning model training tasks.

The request may be expressed as a request including a machine learning model training task and configuration information corresponding to the task. The request may also be expressed as a request including multiple machine learning model training tasks and configuration information corresponding to the tasks. The request may also be represented as a request including only one machine learning model training task or configuration information corresponding to one machine learning model training task. The request may also be represented as a request including multiple machine learning model training tasks or configuration information corresponding to the multiple machine learning model training tasks.

This method is aimed at the situation that in a team-based multi-machine learning model development task, multiple team members need to perform machine learning model training tasks, and at least one of the team members needs to simultaneously perform multiple machine learning model training tasks. Use containerization technology to create a corresponding workspace for each data port, and execute the corresponding machine learning model training tasks in the workspace. Correspondingly, any team member in the team can log in to the model training platform to view other team members The experimental results of the machine learning model. This method takes into account another special situation in the development task of team-based multi-machine learning models, and ensures that the team members can quickly view and compare experimental results, improving the efficiency of team-based development tasks.

In some implementations, the data set for machine learning model training is pre-stored in the model file data set, the training data includes training set data, and the machine learning model training framework is pre-configured in the workspace; The step of executing the machine learning model training task in the workspace includes: obtaining the machine learning model training data from the model file data set according to the configuration information and obtaining the machine learning model code from the text file data set file data; using the training data, the code file data and the training framework to execute the machine learning model training task.

In some cases, for a team-based multi-machine learning model development task, different machine learning models use different training data. For example, the training data used by the machine learning model for image recognition is image data, and the training data used by the machine learning model for speech recognition is voice data. For the same machine learning model, there may also be multiple different versions of the dataset. Therefore, for a team-based development task of multiple machine learning models, the data used for training the machine learning models also needs to be managed in a unified manner. Different machine learning models may use different machine learning model training frameworks. Correspondingly, the code file data used may also be different. Therefore, it is considered to save time for team members to configure corresponding development tools and environments. The machine learning model training framework can be directly preconfigured in the workspace. Correspondingly, the data for machine learning model training is stored in the model file data set in advance, and the code file data of the machine learning model is stored in the text file data set in advance. When needed, it is called from the model file data set and the text file data set.

The data for machine learning model training includes training set data, which can be divided into image data files, text data files, voice data files, video data files, and the like.

The machine learning model training framework is preconfigured, for example, machine learning frameworks such as Pytorch, TensorFlow, XGBoost, and scikit-learn. The code file is a file corresponding to the code of the machine learning model.

This method can greatly save the time for team members to prepare data sets and configure corresponding development tools and environments. Improve team member productivity.

In some implementations, the first type of data also includes machine learning model evaluation index file data; after the step of obtaining the first type of data and the second type of data generated by machine learning model training, it also includes: The evaluation index in the index file data is compared with the preset evaluation index threshold, and if the preset condition is satisfied, the machine learning model training task is continued.

In some cases, the machine learning model needs to undergo multiple rounds of iterative training before the loss function of the machine learning model can converge. After the loss function converges, the obtained machine learning model is not necessarily what is needed for this task. For example, some machine learning model evaluation indicators of this machine learning model may not meet the requirements of this task. Therefore, further training is required to meet the requirements of this task.

The machine learning model evaluation index file data represents the data for evaluating the performance index of the machine learning model. The performance indicators can include accuracy rate (Accuracy), precision rate (Precision), recall rate (Recall), P-R curve (Precision-Recall Curve), false positive rate (FPR), F1 Score and so on.

The threshold of the machine learning model evaluation index can be set independently by humans, and the specific setting needs to correspond to specific tasks. The machine learning model evaluation indicator threshold can be set for all machine learning model evaluation indicators, or only for some key machine learning model evaluation indicators. As long as certain key machine learning model evaluation indicators included in the obtained machine learning model evaluation index file data reach the threshold after a certain machine learning model is trained. You can also stop training. Which machine learning model evaluation indicators to choose as the key machine learning model evaluation indicators can be set corresponding to specific tasks.

The constraint condition may be expressed as continuing to execute the machine learning model training task when a certain evaluation index is lower than the preset evaluation index threshold. It may also be expressed as continuing to execute the machine learning model training task when a certain evaluation index is higher than the preset evaluation index threshold. Specifically, in a specific machine learning training task, the evaluation index threshold of the accuracy rate can be set to 99%, and the constraint condition can be expressed as when the accuracy rate is lower than 99%, continue to Execute the machine learning model training task. In a specific machine learning training task, the evaluation index threshold of the false alarm rate can be set to 1%, and the constraint condition can be expressed as when the false alarm rate is higher than 1%, continue to execute The machine learning model training task.

This method takes into account that in actual application scenarios, the corresponding evaluation index of the machine learning model obtained after training may not meet the actual needs of the task, so further training is required. This method judges whether to perform training again by comparing the machine learning model evaluation index represented by the machine learning model evaluation index file with a preset machine learning model evaluation index threshold. It has obvious use value and can improve the work efficiency of team members.

In some embodiments, the method further includes: receiving an access request for the file index information; in response to the terminal's access request for the file index information, sending the file index information to the terminal for forming An index page for display; wherein, the index page includes the access entry information of the first type of data and the second type of data of the machine learning model or a text identifier bound to the access entry information.

In some cases, there is such a need for a team-based multi-machine learning model development task. Each team member expects to be able to simultaneously query or browse the evaluation indicators, model file data, code file data, etc. obtained after the training of each machine learning model is completed on a terminal page. That is, each team member expects to be able to quickly view and compare the experimental results of other members of the team. Specifically, when team members A and B train the same machine learning model at the same time. After the training of the machine learning model trained by team member A is completed, team member B can quickly query this situation through a front-end page, and then team member B can stop the training. Therefore, if a terminal page is provided to enable all team members to query the experimental results of themselves or other team members, the work efficiency of team members can be improved.

The index page is a page for visualizing the first type of data and the second type of data. This page mainly displays the information contained in the first type of data and the second type of data. The access entry information may be represented as information corresponding to the actual storage address of the first type of data and the second type of data of the machine learning model. A text identifier bound to the access entry information, the text identifier may represent information carried by the first type of data and the second type of data.

Specifically, for example, the first type of data includes model file data of the machine learning model, file data of machine learning model evaluation index files, and data for machine learning model training. The second type of data includes the code file data of the machine learning model, the metadata of the model file, the metadata of the machine learning model evaluation index file data, and the metadata of the machine learning model training data. Then the access entry information may be expressed as the actual storage address of the above-mentioned data. The text identifier bound to the access entry information can be expressed as the information carried by the above data. For example, the information carried in the model file data of the machine learning model includes the type name of the machine learning model, model parameters, and so on. The information carried in the machine learning model evaluation index file data includes various evaluation index values of specific machine learning models. More specifically, the index page is represented as a page obtained by visualizing data included in the model file data of the machine learning model, the machine learning model evaluation index file data, and the code file data of the machine learning model. You can visualize the machine learning model type name, machine learning model evaluation index, and machine learning model code version. The specific visualization form can be in various forms, for example, it can be in the form of a graph, a pie chart, or a table. When a table is used, the header of the table may include training start time, training duration, team members, machine learning model name, machine learning model evaluation index, hyperparameters, and so on.

This method takes into account that, for a team-based multi-machine learning model development task, each team member can query the experimental results of other members of the team on a front-end page, which improves the efficiency of team members' query and comparison of experimental results, and further It has greatly improved the work efficiency of team members and has very good practical value.

As shown in FIG. 6 , an embodiment of this specification provides a method for displaying index information, and the method for displaying index information can be applied to a terminal. The method includes the following steps.

Step S201: Receive index information sent by the model training platform; wherein, the index information is obtained according to one of the above-mentioned methods for generating index information.

Step S202: Form an index page according to the index information; the index page includes access entry information corresponding to the first type of data and the second type of data of the machine learning model or a text identifier bound to the access entry information.

Step S203: Display the index page.

In some cases, there is such a need for a team-based multi-machine learning model development task. Each team member expects to be able to simultaneously query or browse the evaluation indicators, model files, code files, etc. obtained after the training of each machine learning model is completed on a front-end page. Therefore, a method for displaying index information can be provided and applied in a terminal. The terminal receives the index information sent by the server, and forms an index page on the front-end page according to the index information. Team members can use this index page to quickly view and compare experimental results.

In some implementation manners, the index information may be sent from the server to the terminal based on the HTTP protocol or the WebSocket2 protocol. The corresponding front-end system in the terminal can be built using the VUE (progressive framework for building user interface) framework, and the front-end system can use the MVVM (Model-View-ViewModel) architecture pattern. For index information, tools such as echarts and antv, a data visualization chart library, can be used to display.

As shown in FIG. 7 , an embodiment of the present specification also provides an apparatus for generating index information. The device is applied to a model training platform, and the model training platform is used for the training of multiple machine learning models; the device includes: an acquisition unit, which is used to acquire the first type of data and the second type of data generated by machine learning model training data; wherein, the first type of data includes at least the model file data of the machine learning model, and the second type of data includes at least the code file data of the machine learning model and the metadata file data of the model file; a storage unit, which is used to store each The first type of data of the machine learning model is all stored in the model file data set of the model training platform, and the second type of data is all stored in the text file data set of the model training platform; the generation unit is used to The file data set and the text file data set generate file index information; wherein, the file index information includes access entry information of the first type of data and the second type of data corresponding to the machine learning model.

In some embodiments, the generating device further includes: a request receiving unit, configured to receive a request for executing multiple machine learning model training tasks sent by a data port; wherein, the request includes the machine learning model training task Configuration information; a space creation unit, configured to create corresponding workspaces for the plurality of machine learning model training tasks according to the configuration information; a task execution unit, configured to execute the machine learning model training tasks in the workspace .

In some implementations, the generating device further includes: a request receiving unit, configured to receive requests for executing machine learning model training tasks sent by multiple data ports; wherein, the request includes executing multiple machine learning model training tasks and Configuration information of the machine learning model training task; a space creation unit, configured to create a corresponding workspace for the data port according to the configuration information; a task execution unit, configured to execute the machine learning in the workspace Model training tasks.

In some implementations, the data set for machine learning model training is pre-stored in the model file data set, and the data for training includes training set data, and a machine learning model training framework is pre-configured in the workspace. The task execution unit includes: a data acquisition module, which is used to obtain data for machine learning model training from the model file data set and obtain code file data of the machine learning model from the text file data set according to the configuration information; execution module , configured to use the training data, the code file data, and the training framework to execute the machine learning model training task.

In some embodiments, the generating device further includes: a calculation unit, configured to calculate a hash value of the first type of data using a cryptographic hash function; a hash value processing module, configured to convert the hash value Store the metadata file data, and use the hash value as the file name of the first type of data.

In some embodiments, the generating device further includes: a response module, configured to send the file index information to the terminal to form an index page for display in response to a terminal's access request for the file index information ; Wherein, the index page includes the access entry information of the first type of data and the second type of data of the machine learning model or a text identifier bound to the access entry information.

The embodiment of this specification also provides a display device for index information. The display device includes: a receiving unit, configured to receive index information sent by the model training platform; wherein, the index information is obtained by using the aforementioned index information generating method; an index forming unit, configured to form an index page according to the index information The index page includes access entry information corresponding to the first type of data and the second type of data of the machine learning model or a text identifier bound to the access entry information; a display unit configured to display the index page.

As shown in FIG. 8 , the embodiment of this specification also provides an electronic device, which may be an electronic device capable of running application programs, such as a smart phone, a tablet computer, and an e-book. The electronic device in this embodiment may include one or more of the following components: a processor, a network interface, a memory, a non-volatile memory, and one or more application programs, wherein one or more application programs may be stored in a non-volatile The volatile memory is configured to be executed by one or more processors, and one or more programs are configured to execute the methods described in the foregoing method embodiments.

The embodiments of this specification also provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a computer, the computer executes the method for generating index information in any of the above-mentioned embodiments.

The embodiments of this specification also provide a computer program product including instructions, which, when executed by a computer, cause the computer to execute the method for generating index information in any of the above embodiments.

It can be understood that the specific examples herein are only intended to help those skilled in the art better understand the implementation of the present specification, rather than limiting the scope of the present invention.

It can be understood that in various implementations in this specification, the serial numbers of the processes do not mean the order of execution, and the execution order of each process should be determined by its functions and internal logic, rather than by the implementation methods of this specification. The implementation process constitutes any limitation.

It can be understood that the various implementation manners described in this specification can be implemented alone or in combination, which is not limited by the implementation manners of this specification.

Unless otherwise stated, all technical and scientific terms used in the embodiments of this specification have the same meaning as commonly understood by those skilled in the technical field of this specification. The terms used in this specification are only for the purpose of describing specific embodiments, and are not intended to limit the scope of this specification. The term "and/or" used in this specification includes any and all combinations of one or more of the associated listed items. As used in the embodiments of this specification and the appended claims, the singular forms "a", "above" and "the" are also intended to include the plural forms unless the context clearly dictates otherwise.

It can be understood that the processor in the embodiments of this specification may be an integrated circuit chip, which has a signal processing capability. In the implementation process, each step of the above-mentioned method embodiment may be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The above-mentioned processor can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), an off-the-shelf programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the implementation manners of this specification can be realized or executed. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like. The steps of the methods disclosed in conjunction with the embodiments of this specification may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register. The storage medium is located in the memory, and the processor reads the information in the memory, and completes the steps of the above method in combination with its hardware.

It can be understood that the memory in the embodiments of this specification may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the non-volatile memory can be read-only memory (ROM), programmable read-only memory (programmable ROM, PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically erasable programmable read-only memory Memory (EEPROM) or Flash. Volatile memory can be random access memory (RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but not be limited to, these and any other suitable types of memory.

Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of this specification.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method implementation, and will not be repeated here.

In the several implementations provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device implementation described above is only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present specification may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.

If the functions described above are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution in this specification is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various implementation modes of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and other media capable of storing program codes.

The above is only a specific implementation of this specification, but the scope of protection of the present invention is not limited thereto. Anyone familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in this specification. It should be covered within the protection scope of this manual. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (10)

1. A method for generating index information, wherein the method is applied to a model training platform, and the method comprises the following steps:

acquiring first class data and second class data generated by training a machine learning model; the first type of data at least comprises model file data of a machine learning model, and the second type of data at least comprises code file data of the machine learning model and metafile data of a model file;

storing the first type data of each machine learning model into a model file data set of the model training platform, and storing the second type data into a text file data set of the model training platform;

generating file index information according to the model file data set and the text file data set; wherein the file index information includes access entry information for a first class of data and a second class of data of the machine learning model.

2. The method of claim 1, wherein prior to the step of obtaining the first type of data and the second type of data generated by the machine learning model training, further comprising:

receiving a request for executing a plurality of machine learning model training tasks sent by a data port; wherein the request includes configuration information of the machine learning model training task;

Creating corresponding workspaces for the plurality of machine learning model training tasks according to the configuration information;

the machine learning model training task is performed within the workspace.

3. The method of claim 1, wherein prior to the step of obtaining the first type of data and the second type of data generated by the machine learning model training, further comprising:

receiving requests for executing machine learning model training tasks sent by a plurality of data ports; wherein the request includes configuration information for performing a plurality of machine learning model training tasks and the machine learning model training tasks;

creating a corresponding working space for the data port according to the configuration information;

the machine learning model training task is performed within the workspace.

4. A method according to claim 2 or 3, wherein the model file data set has stored therein machine learning model training data in advance, the training data including training set data, the working space having a machine learning model training framework pre-configured therein; the step of performing the machine learning model training task within the workspace comprises:

Acquiring machine learning model training data from the model file data set according to the configuration information and acquiring code file data of a machine learning model from the text file data set;

and executing the machine learning model training task by using the training data, the code file data and the training frame.

5. The method according to claim 1, wherein the method further comprises:

calculating a hash value of the first type of data using a cryptographic hash function;

and storing the hash value into the metadata, and taking the hash value as the file name of the first type of data.

6. The method according to claim 1, wherein the method further comprises:

responding to an access request of a terminal to the file index information, and sending the file index information to the terminal for forming an index page for displaying; the index page comprises access entry information of first-class data and second-class data of the machine learning model or text identification bound with the access entry information.

7. A method for displaying index information, the method comprising:

Receiving index information sent by a model training platform; wherein the index information is obtained by the method of any one of claims 1-6;

forming an index page according to the index information; the index page comprises access entry information of first class data and second class data corresponding to a machine learning model or text identifiers bound with the access entry information;

and displaying the index page.

8. An index information generating device is characterized in that the device is applied to a model training platform; the device comprises:

an acquisition unit for acquiring first-class data and second-class data generated by machine learning model training; the first type of data at least comprises model file data of a machine learning model, and the second type of data at least comprises code file data of the machine learning model and metafile data of a model file;

the storage unit is used for storing the first type data of each machine learning model into a model file data set of the model training platform, and storing the second type data into a text file data set of the model training platform;

a generating unit for generating file index information according to the model file data set and the text file data set; the file index information comprises access entry information of first-class data and second-class data corresponding to the machine learning model.

9. An electronic device comprising a memory and a processor, the memory storing a computer program, the processor implementing the method of generating index information according to any one of claims 1 to 6 or implementing the method of displaying index information according to claim 7 when executing the computer program.

10. A computer-readable storage medium, wherein a computer program is stored in the readable storage medium, and the computer program when executed by a processor implements the method of generating index information according to any one of claims 1 to 6, or implements the method of displaying index information according to claim 7.

Images