CN111949800A - A method and system for establishing a knowledge graph of an open source project - Google Patents

Abstract

The embodiment of the invention provides a method and a system for establishing an open source project knowledge graph, which specifically define a data mode of the open source project knowledge graph in advance; acquiring knowledge information of a program code from an open source project code, acquiring knowledge information related to the open source project from an open source community where the open source project is located and a remote warehouse of the open source project, analyzing the knowledge information of all different sources, and extracting a plurality of triples; unifying and disambiguating all triples, constructing a knowledge graph of the open source project based on the data mode, and finally carrying out visual analysis and display on the knowledge graph. The embodiment of the invention establishes the knowledge graph of the open source project, which is used for a developer to quickly and accurately search the project code to be learned and understand the code through the related code knowledge, thereby meeting the requirement of the newly added developer on quick learning of the open source project.

Description

A method and system for establishing a knowledge graph of an open source project

technical field

The invention relates to the technical field of open source projects, in particular to a method for establishing a knowledge graph of an open source project and a system for establishing a knowledge graph of an open source project.

Background technique

An open source project is an open source software project. Developers can modify the source code of the open source project through the open source community and customize their own personalized products.

Large-scale open source projects are usually developed by multiple developers, and attract many developers to learn the source code of the open source project. After continuous learning and technical experience, these developers may also enter the main open source project. In the branch, contribute your own strength to the open source project.

Most open source projects lack project architecture documents, as well as management and retrieval functions for project code knowledge, while the main functions of the current open source community focus on version management of projects, which are only for existing developers of open source projects and those who do not. Users who contributed to the development. Therefore, when new developers first come into contact with an open source project, they can only understand the project code by reading the source code step by step, and it is difficult to directly find the code related to the requirements, and the learning efficiency is very low.

At present, most of the research in the field of open source community code analysis focuses on analyzing the code itself, and the analysis methods mainly use information such as syntax trees and static analysis results. The research on developer smart learning mostly focuses on how to help developers program smartly, such as recommending code segments, inferring developers' intentions, and smart programming tools. In general, without considering the developer's learning needs, it provides developers with targeted code-related knowledge information to help developers quickly understand and join open source projects.

Therefore, for newly added developers, it is not possible to quickly find the required project code, nor to quickly acquire relevant knowledge of the project code to understand the code, which eventually leads to low learning efficiency.

SUMMARY OF THE INVENTION

In view of the above problems, it is proposed that the embodiments of the present invention provide an open source project knowledge graph establishment method and a corresponding open source project knowledge graph establishment system that overcomes the above problems or at least partially solves the above problems.

In order to solve the above problems, an embodiment of the present invention provides a method for establishing a knowledge graph of an open source project. The method includes: predefining a data schema of the knowledge graph of an open source project; obtaining the program code itself from the code of the open source project through a static code analysis method The knowledge information of the program code itself includes: functions and files; the knowledge information related to the open source project is obtained from the open source community where the open source project is located and the remote warehouse of the open source project. The knowledge information related to the open source project Including: project submission records, code merge requests, and problem sets; analyzing the knowledge information of the program code itself and the knowledge information related to the open source project, extracting multiple triples, and according to the triples Different structural features of each data source in the group, unify the data format of each knowledge entity in all triples, and disambiguate each triple to ensure that the triples are concentrated and each valid knowledge entity There is one and only one entity name corresponding to it; based on the data schema, use the triplet set to construct the knowledge graph of the open source project; use the visualization tool Gephi to visually analyze and display the knowledge graph of the open source project .

Optionally, the method for predefining the data schema of the knowledge graph of an open source project includes: extracting knowledge information constituting the knowledge graph from multiple perspectives of the open source project, including: basic element relationships and entities;

Optionally, the entities include: functions, files, project submission records, issue sets, and code merge requests, and the basic element relationships include: calling relationships, including relationships, modifying relationships, and involving relationships;

Optionally, the method for obtaining the knowledge information of the program code itself from the open source project code includes: obtaining the knowledge information of the program code itself from the open source project code through a static code analysis method; Each file and each project module analyze the open source project code, respectively generate a partial relational subgraph, and the relational subgraph is described and output in Dot language;

Optionally, the static analysis tools include: Doxygen, CppCheck and FindBugs;

Optionally, the multiple triples include: a triple of the calling relationship of the sub function to the obj function, a triple of the inclusion relationship of the sub file to the obj function, and a triple of the modification relationship of the sub submission record to the obj file. , Sub problem set to obj project submission record involving relationship triples, sub problem set to obj problem set involving relationship triples, sub problem set to obj merge request involving relationship triples, sub merge request to obj project The relationship triples involved in the submission record, and the relationship triples involved in the sub code merge request to the obj file

Optionally, the method for constructing a knowledge graph of an open source project by using the multiple triples based on the data schema includes: cleaning and extracting triples for each data source based on the data schema. Group, disambiguate, extract the triplet set representing a certain relationship separately, form a relationship triplet set subgraph, extract the process of constructing the triplet set subgraph concurrently, and finally aggregate all triplet sets Subgraph, to build a knowledge graph of open source projects.

The embodiment of the present invention also provides a system for establishing a knowledge graph of an open source project, the system specifically includes:

Definition module, which is used to predefine the data schema of the knowledge graph of open source projects;

The knowledge acquisition module is used to acquire knowledge information of the program code itself from the open source project code through the static code analysis method, the knowledge information of the program code itself includes: functions, files, and the calling and inclusion relationships between them; and, from Obtain knowledge information related to the open source project from the open source community where the open source project is located and the remote repository of the open source project, where the knowledge information related to the open source project includes: project submission records, code merge requests, and issue sets and the relationships between them relation;

The knowledge analysis module is used to analyze the knowledge information of the program code itself and the knowledge information related to the open source project, and extract a plurality of triples; and according to the difference of each data source in the triples Structural features, unify the data format of each knowledge entity in the triplet, and disambiguate each triplet to ensure that the triplet is concentrated, and each valid knowledge entity has one and only one entity name Corresponding;

a building module for constructing a knowledge graph of an open source project based on the data schema and using the multiple triples;

The display module is used to visually analyze and display the knowledge graph of the open source project by using the visualization tool Gephi.

Optionally, the definition module extracts knowledge information constituting the knowledge graph from multiple perspectives of open source projects, including: basic element relationships and entities; wherein, the entities include: functions, files, project submission records, and problem sets , a code merge request; wherein, the basic element relationship includes: calling relationship, containing relationship, modifying relationship, and involving relationship.

Optionally, the method for predefining the data schema of the knowledge graph of an open source project includes: extracting knowledge information constituting the knowledge graph from multiple perspectives of the open source project, including: basic element relationships and entities;

Optionally, the entities include: functions, files, project submission records, issue sets, and code merge requests, and the basic element relationships include: calling relationships, including relationships, modifying relationships, and involving relationships;

Optionally, the method for obtaining the knowledge information of the program code itself from the open source project code includes: obtaining the knowledge information of the program code itself from the open source project code through a static code analysis method; Each file and each project module analyze the open source project code, respectively generate a partial relational subgraph, and the relational subgraph is described and output in Dot language;

Optionally, the static analysis tools include: Doxygen, CppCheck and FindBugs;

Optionally, the multiple triples include: a triple of the calling relationship of the sub function to the obj function, a triple of the inclusion relationship of the sub file to the obj function, and a triple of the modification relationship of the sub submission record to the obj file. , Sub problem set to obj project submission record involving relationship triples, sub problem set to obj problem set involving relationship triples, sub problem set to obj merge request involving relationship triples, sub merge request to obj project The relationship triples involved in the submission record, and the relationship triples involved in the sub code merge request to the obj file

Optionally, the method for constructing a knowledge graph of an open source project by using the multiple triples based on the data schema includes: cleaning and extracting triples for each data source based on the data schema. Group, disambiguate, extract the triplet set representing a certain relationship separately, form a relationship triplet set subgraph, extract the process of constructing the triplet set subgraph concurrently, and finally aggregate all triplet sets Subgraph, to build a knowledge graph of open source projects.

It can be seen from the above technical solutions that the embodiments of the present invention provide a method and system for establishing a knowledge graph of an open source project. From the open source community where you are located and the multiple data sources of the remote warehouse of the open source project, extract the knowledge information of the project code itself and the knowledge information related to the project code that can be used by developers to learn and participate in the development of open source projects. Knowledge graph, and then comprehensively and effectively display the code knowledge of open source projects, so as to help improve the learning efficiency of developers, encourage developers to better participate in the development of open source projects, and contribute to the development of open source projects.

Description of drawings

1 is a flow chart of steps of an embodiment of a method for establishing a knowledge graph of an open source project provided by the present invention;

Fig. 2 is a kind of SPO triple structure diagram that the embodiment of the present invention provides;

3 is a data schema structure diagram designed for an open source project knowledge graph provided by an embodiment of the present invention;

4 is a partial function call relationship subgraph in a static code analysis result provided by an embodiment of the present invention;

5 is a structural block diagram of an embodiment of a system for establishing a knowledge graph of an open source project provided by the present invention;

6 is a visualization rendering of a knowledge map of an open source project constructed in an embodiment of the present invention;

FIG. 7 is a frame structure diagram of an embodiment of a method for establishing a knowledge graph of an open source project provided by the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Example 1

FIG. 1 is a flow chart of steps of an embodiment of a method for establishing a knowledge graph of an open source project provided by the present invention.

1 , the method for establishing a knowledge graph of an open source project provided by this embodiment is applied to an open source project in an open source community. The relevant knowledge information in the community is used to establish the knowledge graph of the open source project, so as to meet the developer's learning needs for unfamiliar code. The specific knowledge graph establishment method includes the following steps:

Step S101 , predefine the data schema of the knowledge graph of the open source project.

The code of an open source project is written in a specific programming language, and the programming language is a structured language. When a developer wants to understand a specific function in an unfamiliar system, he often cannot just locate and read the function, but also needs to Starting from the outermost layer of the program call relationship, follow the function call to gradually deepen to the target function, and learn and become familiar with the entire function and its positioning in the system through the call path; for example, starting from the relevant functions in the unit test, and deepening step by step Go to the specific function under test to understand the details of the function's call and operation under the test case.

Most of the current analysis research on open source projects is aimed at code search or feature localization. The main purpose of these studies is how to better use open source projects from the user's point of view or how to analyze the project code structure. But for the actual open source community and its contributors and learners, it is very important to be able to participate in the contribution of open source projects with a low threshold. Most open source projects in the current open source community do not maintain system design documents for developers, which means that if a new developer wants to develop a certain function, it needs to spend a lot of time reading and learning the project code. If the organizational structure or program annotation of the project is not perfect, it will cause great difficulties and challenges for developers to participate in actual development. Therefore, according to the developer's learning needs, the present invention constructs the corresponding knowledge map of the open source project symptomatically, so as to help the developer learn unfamiliar codes.

The essence of knowledge graph is a knowledge base of semantic network, which aims to describe the relationship between entities in various kinds of knowledge in the real world and them. An entity can refer to a thing object in reality, or it can be an abstract concept, while a relationship is the relationship between entities and their semantic description. A knowledge graph can usually be viewed as a graph structure, in which the entities of the knowledge graph serve as the nodes of the graph, and the relationships serve as the edges in the graph.

The knowledge graph was first applied to search engines. When a user uses a search engine to search for and understand a piece of knowledge, the search engine can use the knowledge graph to identify the object specifically referred to by the user. For example, when a user searches for the showtime of a drama, the results for the novel of the same name will not be mixed in.

Generally, the knowledge graph can be displayed in the form of a knowledge panel on a web page, which can not only display the website link of the user's search content, but also perform structured aggregation and display of the information of the search topic.

It can be seen that the knowledge graph is composed of some interconnected relationships and their attributes, which are usually represented as an SPO triple (Subject-Predicate-Object). As shown in Figure 2, in a triple, Subject represents the subject, Predicate represents the relationship itself, Object represents the object pointed to by the relationship, and both the subject and the object are entities of the knowledge graph.

The data schema (Schema) in the knowledge graph is the refinement and specification of knowledge. Pre-designing and complying with the given Schema is helpful for standardization and facilitates subsequent processing and querying of knowledge triples. In order to build the knowledge graph of open source projects, We first define the data schema of the knowledge graph. Building a Schema for a knowledge graph is equivalent to building an ontology for a knowledge graph. The ontology includes concepts, concept levels, attributes, attribute value types, relations, relation domain (Domain) concept sets, and relation value domain (Range) concept sets. On this basis, we can also add additional rules (Rules) or axioms (Axioms) to express more complex constraints at the schema layer.

FIG. 3 shows a data schema designed for an open source project knowledge graph provided by an embodiment of the present invention.

The data schema includes extracting basic element relationships and entities constituting the knowledge graph from multiple perspectives of the open source project. Wherein, the entities include: functions, files, project submission records, problem sets, and code merge requests; wherein, the basic element relationships include: calling relationships, inclusion relationships, modification relationships, and involving relationships.

It is worth noting that the embodiments of the present invention are based on the needs of developers to learn open source projects and enable them to participate in open source project development, and the knowledge information extracted from multiple perspectives of open source projects is not the knowledge information required by users on how to use open source projects. , and it is not the code segment recommended knowledge information required by mature developers in smart programming. For example, this application extracts more information inside open source projects, so that developers can understand and contribute code to open source projects and develop them later, instead of extracting API information for developers to use APIs. The embodiment of the present invention extracts the question set of the target open source project in the open source community, that is, the basic element relationship between the knowledge entities and the entities in the discussion text information of the developer on the target open source project, so as to record the submission of functions, files and projects, Information such as issue sets and code merge requests are connected to each other, helping developers to quickly understand and learn the project code and related code knowledge, instead of only extracting the description information of the discussion and the question-and-answer relationship in the discussion. At the same time, those skilled in the art can understand that the knowledge graph provided by the embodiment of the present invention is used to meet the needs of newly added developers for rapid learning and participation in development of open source projects.

Table 1 below is the knowledge graph data schema designed by the embodiment of the present invention, the entities that need to be extracted to construct the knowledge graph and the corresponding descriptions of the entities.

Entities and descriptions included in the knowledge graph described in Table 1

Specifically, it includes: the entity "Func", that is, a function, representing a function in the code of the open source project; the entity "File", that is, a file, representing a file in the open-source project; the entity "Commit", that is, the project submission record , which represents a commit record in the commit history of an open source project; the entity "Issue", that is, a collection of issues, represents a collection of issues and comments of an open source project in the open source community; the entity "Pull Request", that is, a code merge request, A code merge request in the open source community on behalf of an open source project.

Table 2 below is the knowledge graph data schema designed by the embodiment of the present invention, the basic element relationships that need to be extracted to construct the knowledge graph and the corresponding descriptions of the basic element relationships.

Basic element relationships included in the knowledge graph described in Table 2

Specifically, it includes: the relationship "(sub, func_call, obj)", which represents the calling relationship of the sub function to the obj function; the relationship "(sub, file_contain_func, obj)", which represents the inclusion relationship between the sub file and the obj function; the relationship "( sub,commit_change_file,obj)", which indicates the modification relationship between the sub project submission record and the obj file; the relationship "(sub,issue_relate_commit,obj)" indicates the relationship between the sub issue set and the obj project submission record; the relationship "(sub,issue_relate_issue" ,obj)", indicating the relationship between the sub issue set and the obj issue set; the relationship "(sub, issue_relate_pr, obj)", indicating the inclusion relationship between the sub issue set and the obj code merge request; the relationship "(sub, pr_relate_commit, obj) ", indicates the relationship between the sub code merge request and the obj project submission record, and the relationship "(sub, pr_relate_file, obj)" indicates the relationship between the sub code merge request and the obj file.

Taking the inclusion relationship of the sub-issue set to the obj code merge request as an example, in the discussion information in the open source community, the developer may refer to the code merge request of an open source project in a discussion related to an open source project Information, at this time, the set of issues discussed by developers is regarded as the inclusion of the code merge request information.

Taking the relationship between the sub problem set and the obj problem set as an example, in the discussion information in the open source community, the developer may refer to another discussion thread in a discussion related to an open source project. A set of issues discussed by a developer is deemed to have a reference to a set of issues discussed in another discussion thread.

In step S102, the knowledge information of the program code itself is obtained from the open source project code.

In this embodiment, the static code analysis method is used to obtain the knowledge information of the program code itself from the open source project code. The knowledge information of the program code itself includes: functions and files in the project code, as well as the calling and inclusion relationships between them. The invocation and inclusion relationship may specifically be the invocation relationship between the sub function and the obj function, the inclusion relationship between the sub file and the obj function, the inclusion relationship between the sub file and the obj function, and the calling relationship between the sub file and the obj file.

Compared with the dynamic code analysis method, the static code analysis method has the advantages of being fast, general, convenient and less dependent. From the perspective of developers reading and learning open source projects, running an unfamiliar large-scale open source project requires setting the operating environment, consulting documents, and selecting parameters, which is time-consuming and labor-intensive for developers. Using the static analysis method to analyze the code itself is very suitable for developers to read and learn open source projects, and can effectively solve or partially solve the above problems.

Optional, static analysis tools, including: Doxygen, CppCheck and FindBugs.

Further, in this embodiment, Doxygen is selected as a static analysis tool for open source project codes. Doxygen is a multi-language, cross-platform static code analysis tool. It has the advantages of wide language support, good versatility, and diversified analysis content. It can analyze the input project code and extract code information, such as function call relationship, file structure and function attributes, etc. It supports many common languages such as C++, C, Java, Objective-C, Python, etc., and can be used in various operating systems.

Preferably, the static analysis tool Doxygen is used to analyze the open source project code for each file and each project module of the open source project, and generate local relationship subgraphs respectively.

Preferably, the relationship subgraph is described in Dot language, and the knowledge information of the relationship subgraph described in the dot language is mapped to the actual function name to realize output. The Dot language is a textual graphics description language for providing a simple method for describing graphics that can be read by humans and computer programs.

Fig. 4 shows some function call relationship subgraphs in a module in the static code analysis result described and outputted in Dot language using Doxygen static analysis, taking this as an example to illustrate the structure and expression of the relationship subgraph.

In step S103, knowledge information related to the open source project is acquired from the open source community and the remote repository of the open source project.

This embodiment takes the widely used open source community GitHub as an example, uses the GitHub API, combined with Pygithub and Pygit framework tools, to obtain the information in the GitHub open source community and Git warehouse, and extracts the effective and target open source through keywords. Project-related knowledge information.

The knowledge information related to the open source project includes: project submission records, code merge requests, problem sets, and related relationships between them.

Specifically, in the embodiment of the present invention, using the PyGithub framework based on GitHubAPIV3 package implementation, from the GitHub open source community, the knowledge information related to the target open source project is extracted through keywords, including from the discussion information between developers on the project, Extract information about issue sets and code merge requests related to the target open source project. Use the PyGit framework to obtain knowledge information related to the target open source project from the Git repository, including traversing and data extraction of the submission record tree of the target open source project, and obtaining the relevant information of the project submission record.

Among them, GitHub API is an open query interface officially provided by the open source community GitHub, which can provide simple use and access to functions in the open source community. Pygit is a Python encapsulation of libgit. It provides a concise interface to access various attributes in objects and perform various operations on git warehouses. The libgit is a portable C language implementation of the main core methods in the version control system Git of open source projects. By generating a reliable and stable C interface link library, developers can easily use the code to call the API. Implements normal operations in Git. Therefore, libgit can be understood as a shared library of Git. Compared with Git, which is used as an independent program, libgit2 removes complex optimizations and non-core functions.

Step S104, analyze the knowledge information of the program code itself and the knowledge information related to the open source project, and extract a plurality of triples.

On the basis of step 103 and based on heuristic rules, the embodiment of the present invention further analyzes the knowledge information of the program code itself and the knowledge information related to the open source project, and extracts multiple triples. Including: extracting multiple knowledge entities and basic element relationships related to the target open source project from the text information by means of text matching and natural language analysis.

The heuristic rule is widely used in the technical field, and there are many optional types, and the specific usage method of the heuristic rule is not repeated in this embodiment of the present invention. Specifically, ant colony algorithm, neural network algorithm, etc. can be selected.

In the embodiment of the present invention, the ternary relationship in the knowledge graph data schema is formally represented by RDF, that is, the knowledge information of the program code itself and the knowledge information related to the open source project are analyzed and extracted, Convert to multiple triples. The RDF (Resource Description Framework) is a resource description framework customized by W3C (WorldWide Web Consortium), and is a technical specification mainly used to describe and express entities/resources more abundantly.

In a preferred embodiment of the present invention, based on the basic element relationships and entities constituting the knowledge graph included in the data schema in step 101, multiple triples are further obtained, and the categories of the multiple triples include: all The triplet of the calling relationship of the sub function to the obj function, the triplet of the inclusion relationship of the sub file to the obj function, the triplet of the modification relationship of the sub submission record to the obj file, the set of the sub questions to the obj project The relation triples of the commit record, the relation triples of the sub problem set to the obj problem set, the relation triples of the sub problem set to the obj merge request, and the sub merge request to the obj project submission The record involving relation triples, the sub code merge request to the obj file involving relation triples

Step S105, according to the different structural features of each data source in the triplet, unify the data format of each knowledge entity in all triples, and disambiguate each triplet to ensure that the triplet Concentration, each valid knowledge entity has one and only one entity name corresponding to it.

Since the knowledge information of the program code itself and the knowledge information related to the open source project are respectively obtained from the open source project code and the open source community, and the data comes from different sources, the representation, data format, Data consistency is highly likely to differ.

In the embodiment of the present invention, by means of text analysis and heuristic rules, all knowledge entities that may have different prefixes and suffixes are unified and disambiguated by rules. Specifically, it includes: first, verify all the extracted knowledge triples according to the predefined knowledge graph schema, and use natural language analysis methods for triples that do not match the schema, define different mapping rules, and extract different sources. The knowledge information in different formats is mapped to the same formatted data, so that the triples composed of all triples are concentrated, and each valid knowledge entity has one and only one entity name corresponding to it.

For example, for the extraction of file knowledge information, the representations of file names in knowledge from different sources may be absolute paths, relative paths, and individual file names. We need to disambiguate these cases so that the names and relationships of the same files can be Corresponding to the same entity, it is convenient to fuse relationships from different sources.

Step S106, based on the data schema, and using the triplet set, construct a knowledge graph of the open source project.

In the embodiment of the present invention, based on the above steps, a triplet set in which all knowledge information is represented in the form of RDF is obtained, and the unification and disambiguation of knowledge entities are completed. At this time, based on the same knowledge entity, the triples are combined to generate the knowledge graph of the open source project.

Because each triplet needs to be verified with the knowledge information in the existing map before it is fused into the existing map, each verification often requires multiple triples to be verified together. The next triplet can only be added after the fusion is completed, and the efficiency of this map construction method is relatively low.

Therefore, in a preferred embodiment provided by the present invention, based on a pre-defined data pattern, for each data source, cleaning, extracting triples, and disambiguation are performed, and a triple representing a certain type of relationship is separately extracted. set, constitute the triplet set subgraphs of this type of relationship, and concurrently extract the data pipeline constructed by the triplet set subgraphs, that is, the construction of multiple triplet set subgraphs is performed concurrently, and finally all the The triplet set subgraph, builds the knowledge graph of open source projects. Compared with the serial processing in the original construction process, the parallel construction method of the graph in this preferred embodiment is more efficient, and also promotes the unification in development.

Step S107, use the visualization tool Gephi to visually analyze and display the knowledge graph of the open source project.

FIG. 6 shows a visualization effect diagram of an open source project knowledge graph constructed in an embodiment of the present invention.

Through the visualization tool Gephi, the information visualization of the knowledge map of open source projects can be realized, which can help system administrators to find problems such as knowledge loss and inability to align in the knowledge map, so as to further optimize and improve the construction process of the knowledge map.

In addition, based on the visual content, the relevant knowledge information of the open source project can be displayed according to the developer's learning needs, and the relevant map sub-graphs can also be displayed according to the knowledge nodes queried by the developer, so as to provide the developer with the relevant knowledge of the target code that needs to be learned. And give users certain prompt information to optimize the use experience, and add some clicks and events to provide richer information.

As shown in FIG. 7 , the embodiment of the present invention is oriented to the learning needs of developers for open source projects, and extracts data from the open source project code itself, the open source community where the open source project is located, and the remote warehouse of the open source project for developers to learn and participate in open source projects. The knowledge required for project development and the knowledge map of open source projects are constructed to help improve the learning efficiency of developers, so that developers can better participate in the development of open source projects.

It should be noted that, for the sake of simple description, the method embodiments are described as a series of action combinations, but those skilled in the art should know that the embodiments of the present invention are not limited by the described action sequences, because According to embodiments of the present invention, certain steps may be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

Embodiment 2

Referring to FIG. 5, there is shown a structural block diagram of an embodiment of an open source project knowledge graph establishment system according to the present invention. The system can be applied to open source projects. This embodiment aims to obtain open source projects from the perspective of developers learning code requirements. The code knowledge information of the open source community and the relevant knowledge information in the open source community are used to establish the knowledge graph of the open source project, so as to meet the developer's learning needs of the unfamiliar code. The establishment system specifically includes:

The definition module 201 is used to predefine the data schema of the knowledge graph of the open source project.

The essence of knowledge graph is a knowledge base of semantic network, which aims to describe the relationship between entities in various kinds of knowledge in the real world and them. An entity can refer to a thing object in reality, or it can be an abstract concept, while a relationship is the relationship between entities and their semantic description. A knowledge graph can usually be viewed as a graph structure, in which the entities of the knowledge graph serve as the nodes of the graph, and the relationships serve as the edges in the graph.

The knowledge graph was first applied to search engines. When a user uses a search engine to search for and understand a piece of knowledge, the search engine can use the knowledge graph to identify the object specifically referred to by the user. For example, when a user searches for the showtime of a drama, the results for the novel of the same name will not be mixed in.

Generally, the knowledge graph can be displayed in the form of a knowledge panel on a web page, which can not only display the website link of the user's search content, but also perform structured aggregation and display of the information of the search topic.

It can be seen that the knowledge graph is composed of some interconnected relationships and their attributes, which are usually represented as an SPO triple (Subject-Predicate-Object). As shown in Figure 2, in a triple (triple), Subject represents the object in it, Predicate represents the relationship itself, and Object represents the subject pointed to by the relationship.

The data schema (Schema) in the knowledge graph is the refinement and specification of knowledge. Pre-designing and complying with the given Schema is helpful for standardization and facilitates subsequent processing and querying of knowledge triples. In order to build the knowledge graph of open source projects, We first define the data schema of the knowledge graph. Building a Schema for a knowledge graph is equivalent to building an ontology for a knowledge graph. The ontology includes concepts, concept levels, attributes, attribute value types, relations, relation domain (Domain) concept sets, and relation value domain (Range) concept sets. On this basis, we can also add additional rules (Rules) or axioms (Axioms) to express more complex constraints at the schema layer.

FIG. 3 shows a data schema designed for an open source project knowledge graph provided by an embodiment of the present invention.

The data schema includes extracting basic element relationships and entities constituting the knowledge graph from multiple perspectives of the open source project. Wherein, the entities include: functions, files, project submission records, problem sets, and code merge requests; wherein, the basic element relationships include: calling relationships, inclusion relationships, modification relationships, and involving relationships.

It is worth noting that this application is based on the needs of developers to learn open source projects and enable them to participate in the development of open source projects. The knowledge information extracted from multiple perspectives of open source projects is not the knowledge information required by users on how to use open source projects, but also It is not the code segment recommended knowledge information required by mature developers in intelligent programming. For example, this application extracts more information inside open source projects, so that developers can understand and contribute code to open source projects and develop them later, instead of extracting API information and allowing developers to use APIs. In addition, this application extracts the problem set of the target open source project in the open source community, that is, the basic element relationship between knowledge entities and entities in the developer's discussion text information on the target open source project, so as to record functions, files and project submissions, Information such as issue sets and code merge requests are connected to each other, helping developers to quickly understand and learn the project code and related code knowledge, instead of only extracting the description information of the discussion and the question-and-answer relationship in the discussion.

Referring to Table 1, in the knowledge graph data schema designed by the embodiment of the present invention, the entities that need to be extracted to construct the knowledge graph specifically include: an entity "Func", that is, a function, representing a function in the code of an open source project; an entity "File" , that is, a file, representing a file in the open source project; the entity "Commit", that is, the project submission record, representing a commit record in the open source project's submission history; the entity "Issue", that is, the issue collection, representing the open source project in the open source A collection of an issue and its comments in the community; the entity "Pull Request", ie, a code merge request, represents a code merge request for an open source project in the open source community.

Referring to Table 2, in the knowledge graph data schema designed by the embodiment of the present invention, the basic element relationship that needs to be extracted to construct the knowledge graph specifically includes: the relationship "(sub, func_call, obj)", which represents the calling relationship of the sub function to the obj function ;The relationship "(sub,file_contain_func,obj)" indicates the inclusion relationship between the sub file and the obj function; the relationship "(sub,commit_change_file,obj)" indicates the modification relationship between the submission record of the sub project and the obj file; the relationship "(sub, issue_relate_commit, obj)", which indicates the relationship between the sub issue set and the obj project submission record; the relationship "(sub, issue_relate_issue, obj)", which indicates the relationship between the sub issue set and the obj issue set; the relationship "(sub, issue_relate_pr, obj) )", indicating the inclusion relationship between the sub problem set and the obj code merge request; the relationship "(sub, pr_relate_commit, obj)", indicating the involving relationship between the sub code merge request and the obj project submission record, the relationship "(sub, pr_relate_file, obj) ", indicating the relationship between the sub code merge request and the obj file.

Taking the inclusion relationship of the sub-issue set to the obj code merge request as an example, in the discussion information in the open source community, the developer may refer to the code merge request of an open source project in a discussion related to an open source project Information, at this time, the set of issues discussed by developers is regarded as the inclusion of the code merge request information.

Taking the relationship between the sub problem set and the obj problem set as an example, in the discussion information in the open source community, the developer may refer to another discussion thread in a discussion related to an open source project. A set of issues discussed by a developer is deemed to have a reference to a set of issues discussed in another discussion thread.

A knowledge acquisition module 202, configured to acquire knowledge information of the program code itself from the open source project code by a static code analysis method, where the knowledge information of the program code itself includes: functions, files, and calls and inclusion relationships between them; and, Obtain knowledge information related to the open source project from the open source community where the open source project is located and the remote repository of the open source project. The knowledge information related to the open source project includes: project submission records, code merge requests, and issue collections and the relationship between them. relationship is involved.

In this embodiment, the static code analysis method is used to obtain the knowledge information of the program code itself from the open source project code. The knowledge information of the program code itself includes: functions and files in the project code, as well as the calling and inclusion relationships between them. The invocation and inclusion relationship may specifically be the invocation relationship between the sub function and the obj function, the inclusion relationship between the sub file and the obj function, the inclusion relationship between the sub file and the obj function, and the calling relationship between the sub file and the obj file.

Compared with the dynamic code analysis method, the static code analysis method has the advantages of being fast, general, convenient and less dependent. From the perspective of developers reading and learning open source projects, running an unfamiliar large-scale open source project requires setting the operating environment, consulting documents, and selecting parameters, which is time-consuming and labor-intensive for developers. Using the static analysis method to analyze the code itself is very suitable for developers to read and learn open source projects, and can effectively solve or partially solve the above problems.

Optional static analysis tools including: Doxygen, CppCheck and FindBugs.

Further, in this embodiment, Doxygen is selected as a static analysis tool for open source project codes. Doxygen is a multi-language, cross-platform static code analysis tool. It has the advantages of wide language support, good versatility, and diversified analysis content. It can analyze the input project code and extract code information, such as function call relationship, file structure and function attributes, etc. It supports many common languages such as C++, C, Java, Objective-C, Python, etc., and can be used in various operating systems.

Preferably, the static analysis tool Doxygen is used to analyze the open source project code for each file and each project module of the open source project, and generate local relationship subgraphs respectively.

Preferably, the relationship subgraph is described in Dot language, and the knowledge information of the relationship subgraph described in the dot language is mapped to the actual function name to realize output. The Dot language is a textual graphics description language for providing a simple method for describing graphics that can be read by humans and computer programs.

Fig. 4 shows some function call relationship subgraphs in a module in the static code analysis result described and outputted in Dot language using Doxygen static analysis, taking this as an example to illustrate the structure and expression of the relationship subgraph.

This embodiment takes the widely used open source community GitHub as an example, uses the GitHub API, combined with Pygithub and Pygit framework tools, to obtain the information in the GitHub open source community and Git warehouse, and extracts the effective and target open source through keywords. Project-related knowledge information. The knowledge information related to the open source project includes: project submission records, code merge requests, problem sets, and related relationships between them.

Specifically, in the embodiment of the present invention, the PyGithub framework based on the GitHub APIV3 package is used to extract knowledge information related to the target open source project from the GitHub open source community through keywords, including from the discussion information between developers on the project. , which extracts information about issue sets and code merge requests related to the target open source project. Use the PyGit framework to obtain knowledge information related to the target open source project from the Git repository, including traversing and data extraction of the submission record tree of the target open source project, and obtaining the relevant information of the project submission record.

Among them, the GitHub API is an open query interface officially provided by the open source community GitHub, which can provide simple use and access to functions in the open source community. Pygit is a Python encapsulation of libgit. It provides a concise interface to access various attributes in objects and perform various operations on git warehouses. The libgit is a portable C language implementation of the main core methods in the version control system Git of open source projects. By generating a reliable and stable C interface link library, developers can easily use the code to call the API. Implements normal operations in Git. Therefore, libgit can be understood as a shared library of Git. Compared with Git, which is used as an independent program, libgit2 removes complex optimizations and non-core functions.

The knowledge analysis module 203 is used to analyze the knowledge information of the program code itself and the knowledge information related to the open source project, and extract a plurality of triples; Different structural features, unify the data format of each knowledge entity in the triplet, and disambiguate each triplet to ensure that the triplet is concentrated, and each valid knowledge entity has one and only one entity The name corresponds to it.

In the embodiment of the present invention, on the basis of the knowledge acquisition module 202, based on heuristic rules, the knowledge information of the program code itself and the knowledge information related to the open source project are further analyzed, and a plurality of ternary elements are extracted. Group. Including: extracting multiple knowledge entities and basic element relationships related to the target open source project from the text information by means of text matching and natural language analysis.

In a preferred embodiment of the present invention, the ternary relationship in the knowledge graph data schema is formally represented by RDF, that is, the knowledge information of the program code itself and the knowledge information related to the open source project are combined Analysis and extraction are performed and converted into multiple triples. The RDF (Resource Description Framework) is a resource description framework customized by W3C (World Wide Web Consortium), and is a technical specification mainly used to describe and express entities/resources more abundantly.

In a preferred embodiment of the present invention, based on the basic element relationships and entities constituting the knowledge graph included in the data schema of the definition module 201, multiple triples are further obtained, and the categories of the multiple triples include: The triple of the calling relationship of the sub function to the obj function, the triple of the inclusion relationship of the sub file to the obj function, the triple of the modification relationship of the sub submission record to the obj file, the triple of the sub problem set to obj The relation triples of the project submission record, the relation triples of the sub problem set to the obj problem set, the relation triples of the sub problem set to the obj merge request, the sub merge request to the obj project The relationship triples involved in the submission record, and the relationship triples involved in the sub code merge request to the obj file

The heuristic rule is widely used in the technical field, and there are many optional types, and the specific usage method of the heuristic rule is not repeated in this embodiment of the present invention. Specifically, ant colony algorithm, neural network algorithm, etc. can be selected.

Since the knowledge information of the program code itself and the knowledge information related to the open source project are respectively obtained from the open source project code and the open source community, and the data comes from different sources, the representation, data format, Data consistency is highly likely to differ.

In the embodiment of the present invention, by means of text analysis and heuristic rules, all knowledge entities that may have different prefixes and suffixes are unified and disambiguated by rules. Specifically, it includes: first, verify all the extracted knowledge triples according to the predefined knowledge graph schema, and use natural language analysis methods for triples that do not match the schema, define different mapping rules, and extract different sources. The knowledge information in different formats is mapped to the same formatted data, so that the triples composed of all triples are concentrated, and each valid knowledge entity has one and only one entity name corresponding to it.

For example, for the extraction of file knowledge information, the representations of file names in knowledge from different sources may be absolute paths, relative paths, and individual file names. We need to disambiguate these cases so that the names and relationships of the same files can be Corresponding to the same entity, it is convenient to fuse relationships from different sources.

The construction module 204 is configured to construct a knowledge graph of an open source project by using the multiple triples based on the data pattern.

In the embodiment of the present invention, based on the above steps, a triplet set in which all knowledge information is represented in the form of RDF is obtained, and the unification and disambiguation of knowledge entities are completed. At this time, based on the same knowledge entity, the triples are combined to generate the knowledge graph of the open source project.

Because each triplet needs to be verified with the knowledge information in the existing map before it is fused into the existing map, each verification often requires multiple triples to be verified together. The next triplet can only be added after the fusion is completed, and the efficiency of this map construction method is relatively low.

Therefore, in a preferred embodiment of the present invention, based on a pre-defined data pattern, for each data source, cleaning, extracting triples, and disambiguation are performed, and a triplet set representing a certain type of relationship is individually extracted. , constitute the triplet set subgraphs of this type of relationship, and concurrently extract the data pipeline constructed by the triplet set subgraphs, that is, the construction of multiple triplet set subgraphs is performed concurrently, and finally all three Tuple set subgraph to build the knowledge graph of open source projects. Compared with the serial processing in the original construction process, the parallel construction method of the graph in this preferred embodiment is more efficient, and also promotes the unification in development.

The display module 205 is used to visually analyze and display the knowledge graph of the open source project by using the visualization tool Gephi.

FIG. 6 shows a visualization effect diagram of an open source project knowledge graph constructed in an embodiment of the present invention.

Through the visualization tool Gephi, the information visualization of the knowledge map of open source projects can be realized, which can help system administrators to find problems such as knowledge loss and inability to align in the knowledge map, so as to further optimize and improve the construction process of the knowledge map.

In addition, based on the visual content, the relevant knowledge information of the open source project can be displayed according to the developer's learning needs, and the relevant map sub-graphs can also be displayed according to the knowledge nodes queried by the developer, so as to provide the developer with the relevant knowledge of the target code that needs to be learned. And give users certain prompt information to optimize the use experience, and add some clicks and events to provide richer information.

As shown in FIG. 7 , the embodiment of the present invention is oriented to the learning needs of developers for open source projects, and extracts data from the open source project code itself, the open source community where the open source project is located, and the remote warehouse of the open source project for developers to learn and participate in open source projects. The knowledge required for project development and the knowledge map of open source projects are constructed to help improve the learning efficiency of developers, so that developers can better participate in the development of open source projects.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments may be referred to each other.

It should be understood by those skilled in the art that the embodiments of the embodiments of the present invention may be provided as a method, an apparatus, or a computer program product. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media having computer-usable program code embodied therein, including but not limited to disk storage, CD-ROM, optical storage, and the like.

Embodiments of the present invention are described with reference to flowcharts and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the present invention. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing terminal equipment to produce a machine that causes the instructions to be executed by the processor of the computer or other programmable data processing terminal equipment Means are created for implementing the functions specified in the flow or flows of the flowcharts and/or the blocks or blocks of the block diagrams.

These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal equipment to operate in a particular manner, such that the instructions stored in the computer readable memory result in an article of manufacture comprising instruction means, the The instruction means implement the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing terminal equipment, so that a series of operational steps are performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby executing on the computer or other programmable terminal equipment The instructions executed on the above provide steps for implementing the functions specified in the flowchart or blocks and/or the block or blocks of the block diagrams.

Although preferred embodiments of the embodiments of the present invention have been described, additional changes and modifications to these embodiments may be made by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications that fall within the scope of the embodiments of the present invention.

Finally, it should also be noted that in this document, relational terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply these entities or that there is any such actual relationship or sequence between operations. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or terminal device comprising a list of elements includes not only those elements, but also a non-exclusive list of elements. other elements, or also include elements inherent to such a process, method, article or terminal equipment. Without further limitation, an element defined by the phrase "comprises a..." does not preclude the presence of additional identical elements in the process, method, article or terminal device comprising said element.

The method for establishing a knowledge graph of an open source project and a system for establishing a knowledge graph for an open source project provided by the present invention have been described above in detail. In this paper, specific examples are used to illustrate the principles and implementations of the present invention. The description of the embodiment is only used to help understand the method of the present invention and its core idea; meanwhile, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in specific embodiments and application scope. As mentioned above, the contents of this specification should not be construed as limiting the present invention.

Claims (10)

1. A method for establishing an open source project knowledge graph, the method comprising:

predefining a data mode of an open source project knowledge graph;

acquiring knowledge information of a program code from an open source project code by a static code analysis method, wherein the knowledge information of the program code comprises: functions, files and calling and containing relations among the functions and the files;

acquiring knowledge information related to the open source project from an open source community where the open source project is located and a remote warehouse of the open source project, wherein the knowledge information related to the open source project comprises: project submission records, code merge requests, problem sets, and related relationships between them;

analyzing the knowledge information of the program code and the knowledge information related to the open source project, extracting a plurality of triples, unifying the data format of each knowledge entity in all the triples according to different structural characteristics of each data source in the triples, and disambiguating each triplet so as to ensure that the triples are concentrated, and each effective knowledge entity has one entity name corresponding to the effective knowledge entity;

constructing a knowledge graph of the open source project by utilizing the three tuple sets based on the data mode;

and carrying out visual analysis and display on the open source project knowledge graph by using a visualization tool Gephi.

2. The method of claim 1, wherein predefining data patterns of the open source project knowledge-graph comprises:

extracting knowledge information forming a knowledge graph from a plurality of angles of an open source project, wherein the knowledge information comprises: basic element relationships and entities; wherein the entity comprises: functions, files, project submission records, problem sets and code merging requests; wherein the basic element relationships comprise: calling a relationship, containing a relationship, modifying a relationship, referring to a relationship.

3. The method of claim 1, wherein obtaining knowledge information of the program code itself from the open source project code by a static code analysis method comprises:

analyzing the open source project codes by using a static analysis tool aiming at each file and each project module of the open source project, and respectively generating local relation subgraphs, wherein the relation subgraphs are described and output by a Dot language;

the static analysis tool, comprising: doxygen, CppCheck and FindBugs.

4. The method of claim 1, wherein the categories of the plurality of triples comprise:

the method comprises the following steps of calling relation triples of a sub function to obj functions, containing relation triples of sub files to obj functions, modifying relation triples of sub submission records to obj files, relation related triples of sub problem sets to obj item submission records, relation related triples of sub problem sets to obj problem sets, relation related triples of sub problem sets to obj merging requests, relation related triples of sub merging requests to obj item submission records, and relation related triples of sub code merging requests to obj files.

5. The method of claim 1, wherein the step of constructing a knowledge-graph of open source items comprises:

and based on the data mode, cleaning, extracting triples and disambiguating each data source, further respectively extracting a ternary set representing a certain type of relation to form a ternary set subgraph, concurrently extracting a data pipeline for constructing the ternary set subgraph, and finally aggregating all the ternary set subgraphs to construct a knowledge graph of the open source project.

6. A system for establishing an open source project knowledge graph, the system comprising:

the definition module is used for predefining a data mode of the open source project knowledge graph;

the knowledge acquisition module is used for acquiring knowledge information of a program code from an open source project code by a static code analysis method, wherein the knowledge information of the program code comprises the following steps: functions, files and calling and containing relations among the functions and the files; and acquiring knowledge information related to the open source project from the open source community where the open source project is located and a remote warehouse of the open source project, wherein the knowledge information related to the open source project comprises: project submission records, code merge requests, and problem sets and the relationship of involvement between them;

the knowledge analysis module is used for analyzing the knowledge information of the program code and the knowledge information related to the open source project and extracting a plurality of triples; unifying the data format of each knowledge entity in the triples according to different structural characteristics of each data source in the triples, and disambiguating each triplet so as to ensure that each effective knowledge entity has one entity name corresponding to the effective knowledge entity in the triples;

a construction module for constructing a knowledge graph of the open source item using the plurality of triples based on the data schema;

and the display module is used for carrying out visual analysis and display on the open source project knowledge graph by using a visualization tool Gephi.

7. The system of claim 6, wherein the definition module comprises:

extracting knowledge information forming a knowledge graph from a plurality of angles of an open source project, wherein the method comprises the following steps: basic element relationships and entities; wherein the entity comprises: functions, files, project submission records, problem sets and code merging requests; wherein the basic element relationships comprise: calling a relationship, containing a relationship, modifying a relationship, referring to a relationship.

8. The system of claim 6, wherein the knowledge acquisition module comprises:

analyzing the open source project codes by using a static analysis tool aiming at each file and each project module of the open source project, and respectively generating local relation subgraphs, wherein the relation subgraphs are described and output by a Dot language;

the static analysis tool, comprising: doxygen, CppCheck and FindBugs.

9. The system of claim 6, wherein in the knowledge analysis module, the categories of the plurality of triples comprise:

the method comprises the following steps of calling relation triples of a sub function to obj functions, containing relation triples of sub files to obj functions, modifying relation triples of sub submission records to obj files, relation related triples of sub problem sets to obj item submission records, relation related triples of sub problem sets to obj problem sets, relation related triples of sub problem sets to obj merging requests, relation related triples of sub merging requests to obj item submission records, and relation related triples of sub code merging requests to obj files.

10. The system of claim 6, the build module, comprising:

and based on the data mode, cleaning, extracting triples and disambiguating each data source, further respectively extracting a ternary set representing a certain type of relation to form a ternary set subgraph, concurrently extracting a data pipeline for constructing the ternary set subgraph, and finally aggregating all the ternary set subgraphs to construct a knowledge graph of the open source project.

Images