WO2022267976A1 - Entity alignment method and apparatus for multi-modal knowledge graphs, and storage medium - Google Patents

Abstract

Disclosed are an entity alignment method and apparatus for multi-modal knowledge graphs, and a storage medium. The present invention comprises: acquiring data of a first multi-modal knowledge graph and a second multi-modal knowledge graph, and extracting therefrom entities that require alignment; processing multi-modal data of the entities to obtain each modal vector of the entities, and performing early fusion and late fusion according to each modal vector; combining the result of early fusion and the result of late fusion to obtain a multi-modal embedded vector; and performing entity alignment according to the multi-modal embedded vector. By using the method of the present invention, entity alignment for multi-modal knowledge graphs can be implemented, thus solving the problem of inconsistency between multi-modal knowledge expressions. The present invention can be widely applied to the technical field of knowledge graphs.

Description

Entity alignment method, device and storage medium for multimodal knowledge graph technical field

The present invention relates to the technical field of knowledge graphs, in particular to an entity alignment method, device and storage medium for multimodal knowledge graphs.

Background technique

Because most knowledge graphs are built for specific purposes and based on a monolingual environment, this leads to the situation that the same concept may appear differently in different knowledge graphs. The purpose of entity alignment is to filter out entities that are different but actually the same in the two knowledge graphs, so as to integrate different knowledge graphs.

Due to the variety of knowledge forms, the current embedding technology cannot handle multi-modal knowledge well. To overcome this challenge, researchers have recently proposed various models to fuse multi-modal information in knowledge graphs and form joint embeddings. Allows the alignment model to automatically adjust the modal weights. However, these studies have not considered the modal correlation at the feature level, and it is likely that unsatisfactory results will be obtained when the correlation between multiple modalities is relatively large. These problems existing in the prior art need to be solved urgently.

Contents of the invention

The purpose of the present invention is to solve one of the technical problems in the prior art at least to a certain extent.

To this end, an object of the embodiments of the present invention is to provide an entity alignment method, device, and medium for multimodal knowledge graphs, which can realize the integration of multimodal knowledge graphs by performing early fusion and late fusion on multimodal knowledge graphs. Entity alignment, which resolves the inconsistency between multimodal knowledge representations.

In order to achieve the above technical objectives, the technical solutions adopted in the embodiments of the present invention include:

In the first aspect, an embodiment of the present invention provides a method for entity alignment of a multimodal knowledge graph, including the following steps:

The entity alignment method of the multimodal knowledge graph is characterized in that it comprises the following steps:

Obtain data of the first multimodal knowledge graph and the second multimodal knowledge graph;

Extract entities to be aligned from the first multimodal knowledge graph and the second multimodal knowledge graph respectively;

Processing the multimodal data of the entity to obtain the modal vectors of the entity, wherein the multimodal data includes image data, relationship data, attribute data, and knowledge map structure data; the modal vectors include Image embedding vector, relationship embedding vector, attribute embedding vector and knowledge map structure vector;

Carry out early fusion through the fully connected neural network model according to the modal vectors;

performing late fusion through a low-rank multimodal model according to each of the modality vectors;

Combining the results of early fusion and late fusion to obtain a multimodal embedding vector;

Entity alignment is performed based on the multimodal embedding vectors.

Further, the step of processing the image data of the entity to obtain the image embedding vector of the entity specifically includes:

Using a pre-trained RESNET model to perform feature extraction on the acquired image data;

The extracted features are processed by a first preset function to obtain an image embedding vector.

Further, the step of processing the relationship data of the entity to obtain the relationship embedding vector of the entity specifically includes:

Converting the obtained relationship data into a translation vector through a TransE model;

Calculating the structural similarity of the translation vector through a second preset function to obtain a logistic regression loss function;

By converging the logistic regression loss function, a relationship embedding vector is obtained.

Further, the step of processing the attribute data of the entity to obtain the attribute embedding vector of the entity specifically includes:

The acquired attribute data is mapped to a low-dimensional space through a feed-forward network to obtain an attribute embedding vector.

Further, the step of processing the knowledge map structure data of the entity to obtain the structure embedding vector of the entity specifically includes:

Building a semi-supervised embedding model based on graph convolutional networks;

set relationship vertices;

Process the relationship vertices through the semi-supervised embedding model to obtain a structure embedding vector.

Further, the early fusion specifically includes:

Establish a fully connected neural network model;

All the features extracted by the RESNET model are fused through the fully connected neural network model.

Further, the late fusion specifically includes:

Simplify the vector representation of multimodal fusion through a low-rank multimodal fusion model;

Simplify the vector representation in a preset manner.

Further, the step of combining the early fusion and the late fusion specifically includes:

The early fusion and the late fusion are combined through collaborative training according to a preset loss function.

In the second aspect, an embodiment of the present invention provides an entity alignment device for a multimodal knowledge graph, including:

at least one processor;

at least one memory for storing at least one program;

When the at least one program is executed by the at least one processor, the at least one processor is made to implement the entity alignment method of the multi-modal knowledge graph.

In the third aspect, the embodiment of the present invention provides a storage medium, which stores processor-executable instructions, and the processor-executable instructions are used to implement the multi-modal knowledge graph when executed by the processor The entity alignment method for .

The invention discloses an entity alignment method of a multimodal knowledge graph, which has the following beneficial effects:

The present invention obtains the data of the first multi-modal knowledge map and the second multi-modal knowledge map, extracts entities that need to be aligned; and then processes the multi-modality composed of image data, relational data, attribute data and knowledge map structure data Entity data, get the modal vectors composed of image embedding vectors, relational embedding vectors, attribute embedding vectors, and knowledge map structure vectors, and perform early fusion and late fusion according to each modal vector; then, the results of early fusion and The results of the late fusion are combined to obtain multimodal embedding vectors; finally, entity alignment is performed based on the multimodal embedding vectors. By using the method in the present invention, the entity alignment of the multi-modal knowledge map can be realized, and the problem of inconsistency between multi-modal knowledge representations can be solved.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following describes the accompanying drawings of the embodiments of the present invention or the related technical solutions in the prior art. It should be understood that the accompanying drawings in the following introduction are only In order to clearly describe some embodiments of the technical solutions of the present invention, those skilled in the art can also obtain other drawings based on these drawings without creative work.

FIG. 1 is a schematic flow diagram of an entity alignment method of a multimodal knowledge graph according to a specific embodiment of the present invention;

Fig. 2 is a flow chart of the application process of an entity alignment method of a multimodal knowledge graph according to a specific embodiment of the present invention;

Fig. 3 is a schematic structural diagram of an entity alignment device for a multi-modal knowledge graph according to a specific embodiment of the present invention.

detailed description

Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary only for explaining the present invention and should not be construed as limiting the present invention. For the step numbers in the following embodiments, it is only set for the convenience of illustration and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art sexual adjustment.

Entity alignment is a key task to integrate different knowledge graphs by arranging various entities of the same real-world prototype, because most knowledge graphs are built for a specific purpose and based on a monolingual environment, resulting in various Even for the same concept, there will be differences in different descriptions.

Much of the early research on entity alignment focused on attribute similarity. These studies are often puzzled by attribute heterogeneity that makes entity alignment error-prone. Recently, given the rapid development of knowledge graph embedding, many researchers have tried to apply embedding techniques based on various models of the entity alignment problem. However, these embedding techniques are not yet able to handle multi-modal knowledge well, because knowledge forms are diverse, such as relational triples, images, etc., but at the same time, these knowledge forms are highly supportive of entity alignment work.

The impact of multimodal knowledge generating entity alignment problems is not trivial, since the inevitable heterogeneity in different modalities makes it difficult to learn and fuse knowledge representations from different modalities. Involving the same target, it is not easy to identify using traditional technology and only using image or text information. To overcome this challenge, researchers have recently proposed various models to fuse multimodal information in knowledge graphs to form a joint embedding, allowing the alignment model to automatically adjust the modal weights. However, these studies did not consider the modal correlation at the feature level, and when the correlation between multiple modalities is relatively large, it is likely that unsatisfactory results will be obtained.

Based on the above problems, this scheme proposes an entity alignment method for multimodal knowledge graphs. This scheme first processes the multi-modal data composed of image data, relational data, attribute data and knowledge map structure data in the entity to obtain the image embedding vector, relational embedding vector, attribute embedding vector and knowledge map structure vector. The modal vector, and then perform early fusion and late fusion according to each modal vector, and then combine the results of early fusion and late fusion to obtain a multimodal embedding vector, so as to solve the problem of multimodal The influence produced when the correlation is relatively large improves the accuracy of entity alignment results.

Specifically, referring to Fig. 1 and Fig. 2, the entity alignment method of the multimodal knowledge map provided by the embodiment of the present invention includes the following steps:

Step 101, acquiring data of a first multimodal knowledge graph and a second multimodal knowledge graph. Knowledge map is a combination of theories and methods of applied mathematics, graphics, information visualization technology, information science and other disciplines with metrology citation analysis, co-occurrence analysis and other methods, and uses the visual map to vividly display the core structure of the subject, Modern theories that develop history, frontier fields, and overall knowledge structure to achieve multidisciplinary integration. Among them, the main difference between the multi-modal knowledge graph and the traditional knowledge graph is that the traditional knowledge graph mainly focuses on the entities and relationships of texts and databases, while the multi-modal knowledge graph constructs a variety of models based on the traditional knowledge graph. modal entities, as well as the multimodal semantic relationship between entities of various modalities.

Step 102, extract entities to be aligned from the multimodal knowledge graph. The modal knowledge graph in this step refers to the first multimodal knowledge graph and the second multimodal knowledge graph in step 101 . The specific operation process refers to extracting entities to be aligned from the first multi-modal knowledge graph and the second multi-modal knowledge graph respectively. Entities are things that exist objectively and can be distinguished from each other, and often refer to a collection of certain types of things.

Step 103, processing the multi-modal data of the entity to obtain the vectors of each mode of the entity. Among them, multimodal data includes image data, relational data, attribute data, and knowledge graph structure data; each modality vector includes image embedding vector, relational embedding vector, attribute embedding vector, and knowledge graph structure vector.

Among them, the image embedding is specifically to use the pre-trained RESNET model as the feature extractor of the image, and take the output of the last layer as the image representation. Finally, the extracted features are processed by the first preset function to obtain the image embedding vector emb_I. Among them, the RESNET model refers to the residual network, which is a kind of convolutional neural network. It is characterized by easy optimization and the ability to increase accuracy by adding considerable depth. Its internal residual block uses skip connections, which alleviates the problem of gradient disappearance caused by increasing depth in deep neural networks. Compared with VGG16, another classic convolutional neural network model, RESNTET can solve the degradation problem in deep networks.

The first default function is as follows:

emb _I ＝W _I *RESNET(I)+b _I

In the above formula, W _I is the weight vector, b _I is the bias vector, and I represents the image.

Among them, relational embedding is specifically, using the TransE model to represent all entities and relations in the multimodal knowledge map as a low-dimensional vector. The role of the TransE model is to translate triples into embedding word vectors. And the triplet, that is, the form of (head entity, relationship, tail entity), the head entity and the tail entity are collectively referred to as entities. For simplicity, f(h,r,t) is used to represent the triplet, h is the head entity, t is the tail entity, and r is the relationship between h and t. Next, a second preset function is used to measure the similarity of the structure.

The second preset function is as follows:

f _rel (h,r,t)＝-||h ⁽²⁾ +rt ⁽²⁾ ||

Among them, f _rel (h, r, t) is a function to calculate the similarity between entity h and entity t.

Thus, the logistic regression loss function is obtained, and the relationship embedding vector emb_r is obtained through the convergence function, as follows:

In the above formula, a is the label of f _rel (h, r, t), and its value is 1 or -1, X ⁺ indicates the positive correlation fact in the source knowledge graph and the target knowledge graph, and X ^- refers to the positive correlation fact by replacing Fact head or tail entities to represent a set of negative samples.

Among them, attribute embedding is specifically, due to the existence of noise from neighboring nodes, using a deep neural network model to process attribute embedding is not effective, so a simple feedforward network is used to map attribute features into a low-dimensional space to obtain attribute Embedding vector:

emb _A =W _A *A+b _A

In the above formula, emb _A is the attribute embedding vector, W _A is the weight matrix vector, b _A is the bias vector, and A is the set of attributes.

Among them, the knowledge map structure embedding is specifically to establish a semi-supervised embedding model based on graph convolutional network, and transform the knowledge map into an undirected graph. The structure of the original knowledge graph is reconstructed. For example, assuming a triplet (e1, r, e2), e1, e2 represent entities, r represents the relationship between entities, and in this embodiment, the semi-supervised embedding model assigns different relationship vertices r1 and r2 to triplets , forming (e1,r1) and (e2,r2). Each relation vertex adopts a unique one-hot representation.

Based on this new undirected graph, the Deepwalk algorithm is used to represent the feature vector of each entity vertex, and the unique one-hot representation of each relationship vertex is input to the GCN system. These relationship vertices can display the total number of neighbors with the same relationship information between two entity vertices. After encoding through the convolutional layer, the representation information of entity vertices and relational vertices in the graph can be obtained. For each layer in GCN can be written as a non-linear function:

H ^(l+1) = f(H ^(l) , M)

In the above formula, H ^(l+1) is the input matrix, H ^(l) is the output matrix, L is the number of layers, and M is the adjacency matrix of the knowledge map. Then, set the propagation rules as follows:

f(H ^(l) ，M)＝ReLU(MH ^(l) W ^(l) )

In the above formula, W ^(l) is the weight matrix of the L network layer, and ReLU is the activation function. Note that multiplying by M only sums all attributes of all adjacent vertices, not the vertices themselves. Therefore, it is necessary to add the identity matrix I to M, so the above equation is updated as follows:

In the above formula, M=M+I, and D is a diagonal matrix of M. In this embodiment, the output of the last layer is used as the structural embedding vector emb_kg of the knowledge graph.

Step 104, performing early fusion through the fully connected neural network model. Early fusion refers to better capturing the relationship in features by combining features before data is fed into the model. This scheme uses standard early fusion techniques to fuse multiple features from different data modalities. In this embodiment, a simple fully connected neural network model is designed to connect all the features of each modality in series.

定义为M个不同模态的单模态信息的编码，多模态融合的目标是将单模态表示集成到一个紧凑的多模态表示中。张量表示被认为多模态融合的一个有效办法。但是，学习权重张量的参数量也将成指数增加。这不仅增多了大量的计算，还使模型有过拟合的风险。本实施例通过低秩多模态融合模型把权重分解为一系列低秩因子集。其中，低秩多模态融合模型可以将

简化为输出向量h _l: Step 105, performing late fusion through the low-rank multimodal model. Bundle

Defined as the encoding of unimodal information from M different modalities, the goal of multimodal fusion is to integrate unimodal representations into a compact multimodal representation. Tensor representations are considered an efficient approach for multimodal fusion. However, the number of parameters of the learned weight tensor will also increase exponentially. This not only increases a large number of calculations, but also puts the model at risk of overfitting. In this embodiment, the weight is decomposed into a series of low-rank factor sets through the low-rank multimodal fusion model. Among them, the low-rank multimodal fusion model can combine

Simplifies to the output vector h _l :

表示一系列张量的元素点积,r是张量的秩，

是每个模态m的相应低秩因子。和现有的方法相比，这个计算方式简化了Z和W的并行分解。这样只需要计算h _l而无需创建张量Z，避免计算大输入张量Z。若r太大，计算量仍然很大。此时，通过交换求和顺序和按元素乘积的方式更新为下列等式： In the above formula,

Represents the element dot product of a series of tensors, r is the rank of the tensor,

is the corresponding low-rank factor for each mode m. Compared with existing methods, this calculation method simplifies the parallel decomposition of Z and W. This way only h _l needs to be calculated without creating tensor Z, avoiding the calculation of large input tensor Z. If r is too large, the amount of calculation is still very large. In this case, update to the following equation by swapping the order of summation and element-wise product:

In the above formula, i represents the i-th item of the matrix, and the newly added constraints are to ensure that the decomposition exists within an acceptable range while reducing the amount of calculation.

Step 106, combining the results of the early fusion and the late fusion to obtain a multimodal embedding vector. Specifically, the late fusion result h _l and the early fusion model he produced are combined by the following loss function to obtain the final multimodal embedding _embF . In this way, the advantages of the two fusions can be combined: not only can the output features of the previous fusion be easily combined, but also the calculations generated by the input tensor process can be avoided, reducing the complexity of calculations.

Step 107, perform entity alignment according to the multimodal embedding vector.

In some embodiments, embedding of multimodal vectors is achieved through multiple trainings. Specifically, all entity embeddings are constrained with an L2 norm to adjust the embedding vectors. The parameters are initialized with the Xavier initializer, and the loss function is optimized with Adadelta to simplify the calculation. In addition to the embF of all entities, it is also necessary to calculate the similarity of all even-graph entity pairings and rank them with the loss function L _ea . L _ea looks like this:

where α and β are temperature scales; N is the number of seeds.

When the entire training process converges, entity alignment is performed through the nearest neighbor search algorithm based on embF.

Provide the concrete experimental data of this embodiment below:

The main content of this experiment is to measure the similarity between two public multi-mode data sets FB15K-DB15K and FB15K-YAGO15K, so as to obtain the performance of this embodiment. In this embodiment, cosine similarity is used to calculate the similarity between two data sets, and Hits@n, MR and MRR are used as indicators for evaluating all models. Hits@n represents the ratio of correct entities ranked in the top n based on similarity calculations. MR indicates the average rank of correct entities. MRR represents the mean reciprocal of correct entities.

Various types of state-of-the-art models are selected in the experiments to demonstrate the performance of the proposed (DFMKE) framework, including two typical translation-based methods, TransE and IPTransE. Two simple late fusion methods: MMKG and MMEA; and two more recent methods: MultiKE and EVA. For those methods that used the same dataset as in this example, the reported results were adopted directly. For other methods, experiments with other methods are repeated following the same hyperparameter settings mentioned in the original paper.

It can be seen from the above table that among the three indicators of Hits@1, Hits@10, and MRR, this embodiment (DFMKE) ranks the highest; among the MR indicators, this embodiment (DFMKE) ranks the lowest. That is to say, this embodiment (DFMKE) has a higher entity alignment accuracy rate than other existing technologies, and effectively solves the problem of inconsistency between multimodal knowledge representations.

Referring to FIG. 3 , an embodiment of the present invention provides an entity alignment device for a multimodal knowledge graph, including:

at least one processor 201;

at least one memory 202 for storing at least one program;

When the at least one program is executed by the at least one processor 201, the at least one processor 201 is enabled to implement the entity alignment method of the multimodal knowledge graph shown in FIG. 1 .

The content in the above-mentioned method embodiment is applicable to this device embodiment, and the specific functions realized by this device embodiment are the same as those of the above-mentioned method embodiment, and the beneficial effects achieved are also the same as those achieved by the above-mentioned method embodiment.

The embodiment of the present invention also provides a storage medium, which stores processor-executable instructions, and the processor-executable instructions are used to realize the entity of the multimodal knowledge graph shown in FIG. 1 when executed by the processor. alignment method.

In some alternative implementations, the functions/operations noted in the block diagrams may occur out of the order noted in the operational diagrams. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality/operations involved. Furthermore, the embodiments presented and described in the flowcharts of the present invention are provided by way of example in order to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logical flow presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are performed independently.

Furthermore, although the invention has been described in the context of functional modules, it should be understood that one or more of the described functions and/or features may be integrated into a single physical device and/or unless stated to the contrary. or software modules, or one or more functions and/or features may be implemented in separate physical devices or software modules. It will also be appreciated that a detailed discussion of the actual implementation of each module is not necessary to understand the present invention. Rather, given the attributes, functions and internal relationships of the various functional blocks in the devices disclosed herein, the actual implementation of the blocks will be within the ordinary skill of the engineer. Accordingly, those skilled in the art can implement the present invention set forth in the claims without undue experimentation using ordinary techniques. It is also to be understood that the particular concepts disclosed are illustrative only and are not intended to limit the scope of the invention which is to be determined by the appended claims and their full scope of equivalents.

If the functions described above are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc., which can store program codes. .

The logic and/or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced listing of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium, For use with instruction execution systems, devices, or devices (such as computer-based systems, systems including processors, or other systems that can fetch instructions from instruction execution systems, devices, or devices and execute instructions), or in conjunction with these instruction execution systems, devices or equipment for use. For the purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use in or in conjunction with an instruction execution system, device or device.

More specific examples (non-exhaustive list) of computer-readable media include the following: electrical connection with one or more wires (electronic device), portable computer disk case (magnetic device), random access memory (RAM), Read Only Memory (ROM), Erasable and Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Compact Disc Read Only Memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program can be printed, since the program can be read, for example, by optically scanning the paper or other medium, followed by editing, interpretation or other suitable processing if necessary. processing to obtain the program electronically and store it in computer memory.

It should be understood that various parts of the present invention can be realized by hardware, software, firmware or their combination. In the embodiments described above, various steps or methods may be implemented by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one or combination of the following techniques known in the art: Discrete logic circuits, ASICs with suitable combinational logic gates, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

In the above description of this specification, the description with reference to the terms "one embodiment/example", "another embodiment/example" or "some embodiments/example" means that the description is described in conjunction with the embodiment or example. Specific features or characteristics are included in at least one embodiment or example of the invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the described specific features or characteristics may be combined in any suitable manner in any one or more embodiments or examples.

Although the embodiments of the present invention have been shown and described, those skilled in the art can understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principle and spirit of the present invention. The scope of the invention is defined by the claims and their equivalents.

The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments, and those skilled in the art can also make various equivalent deformations or replacements without violating the spirit of the present invention. These equivalent modifications or replacements are all within the scope defined by the claims of the present application.

Claims (10)

The entity alignment method of the multimodal knowledge graph is characterized in that it comprises the following steps: Obtain data of the first multimodal knowledge graph and the second multimodal knowledge graph; Extract entities to be aligned from the first multimodal knowledge graph and the second multimodal knowledge graph respectively; Processing the multimodal data of the entity to obtain the modal vectors of the entity, wherein the multimodal data includes image data, relationship data, attribute data, and knowledge map structure data; the modal vectors include Image embedding vector, relationship embedding vector, attribute embedding vector and knowledge map structure vector; Carry out early fusion through the fully connected neural network model according to the modal vectors; performing late fusion through a low-rank multimodal model according to each of the modality vectors; Combining the results of early fusion and late fusion to obtain a multimodal embedding vector; Entity alignment is performed based on the multimodal embedding vectors. The entity alignment method of the multimodal knowledge map according to claim 1, wherein the step of processing the image data of the entity to obtain the image embedding vector of the entity specifically includes: Using a pre-trained RESNET model to perform feature extraction on the acquired image data; The extracted features are processed by a first preset function to obtain an image embedding vector. The entity alignment method of the multimodal knowledge map according to claim 1, wherein the step of processing the relational data of the entity and obtaining the relational embedding vector of the entity specifically includes: Converting the obtained relationship data into a translation vector through a TransE model; Calculating the structural similarity of the translation vector through a second preset function to obtain a logistic regression loss function; By converging the logistic regression loss function, a relationship embedding vector is obtained. The entity alignment method of the multimodal knowledge map according to claim 1, wherein the step of processing the attribute data of the entity to obtain the attribute embedding vector of the entity specifically includes: The acquired attribute data is mapped to a low-dimensional space through a feed-forward network to obtain an attribute embedding vector. The entity alignment method of the multimodal knowledge graph according to claim 1, wherein the step of processing the knowledge graph structure data of the entity to obtain the structural embedding vector of the entity specifically includes: Building a semi-supervised embedding model based on graph convolutional networks; set relationship vertices; Process the relationship vertices through the semi-supervised embedding model to obtain a structure embedding vector. The entity alignment method of the multimodal knowledge map according to claim 2, wherein the early fusion specifically includes: Establish a fully connected neural network model; All the features extracted by the RESNET model are fused through the fully connected neural network model. The entity alignment method of the multimodal knowledge map according to claim 1, wherein the late fusion specifically includes: Simplify the vector representation of multimodal fusion through a low-rank multimodal fusion model; Simplify the vector representation in a preset manner. The entity alignment method of the multimodal knowledge map according to claim 1, wherein the step of combining the early fusion and the late fusion specifically includes: The early fusion and the late fusion are combined through collaborative training according to a preset loss function. The entity alignment device of the multimodal knowledge graph is characterized in that it includes: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the entity alignment method of the multi-modal knowledge graph according to any one of claims 1-8. A computer-readable storage medium, in which processor-executable instructions are stored, wherein the processor-executable instructions are used to implement any one of claims 1-8 when executed by a processor. An entity alignment method for the multimodal knowledge graph described above.

Images