WO2017135889A1

WO2017135889A1 - Ontology determination methods and ontology determination devices

Info

Publication number: WO2017135889A1
Application number: PCT/SG2016/050062
Authority: WO
Inventors: Abeykoon Mudiyanselage Hunfuko Asanka Abeykoon; Wujuan Lin; Kazuya Monden
Original assignee: Hitachi Ltd
Current assignee: Hitachi Ltd
Priority date: 2016-02-05
Filing date: 2016-02-05
Publication date: 2017-08-10
Anticipated expiration: 2018-08-05

Abstract

According to various embodiments, an ontology determination method may be provided. The ontology determination method may include: determining a first ontology; determining additional ontology information; determining whether the first ontology is to be merged with the additional ontology information based on external information including at least one of search preferences, user preference, social networking search engine data, web search engine data, or external ontologies; and determining a second ontology based on the first ontology and the additional ontology information if it is determined that the first ontology is to be merged with the additional ontology information.

Description

ONTOLOGY DETERMINATION METHODS AND ONTOLOGY

DETERMINATION DEVICES

Technical Field

[0001] Embodiments relate generally to ontology determination methods and ontology determination devices.

Background

[0002] There are two main categories of previously used methods where one category provides an automatically evolving ontology without controls while the other category strictly controls such ontology.

[0003] US 2013/0282747 and EP 2,204,747 Al allow automatically evolving ontology which may cause accuracy issues and performance issues which are previously discussed. In such cases, ontology may evolve expectedly larger using internal and/or external data, information or sources. Thus, erroneous search results and performance issues can be considered as major limitations arise in this category.

[0004] US 8,867,891 B2 and US 2010/0223223 Al rather control the ontology more strictly. In such cases, semantic classifiers are more predefined. In other words, only predefined sets of concepts may be analyzed and thus the ontology evolving may be more limited than preferred. Thus, in this category, there may be a gap in query space and concepts which are being analyzed or for which metadata is generated. However, this gap may further cause giving unrelated search results or inaccurate search results. [0005] As such, previously used methods fail to provide an appropriate ontology control. Thus, there may be a need for more enhanced methods.

Summary

[0006] According to various embodiments, an ontology determination method may be provided. The ontology determination method may include: determining a first ontology; determining additional ontology information; determining whether the first ontology is to be merged with the additional ontology information based on external information including at least one of search preferences, user preference, social networking search engine data, web search engine data, or external ontologies; and determining a second ontology based on the first ontology and the additional ontology information if it is determined that the first ontology is to be merged with the additional ontology information.

[0007] According to various embodiments, an ontology determination device may be provided. The ontology determination device may include: a first ontology determination circuit configured to determine a first ontology; an additional ontology information determination circuit configured to determine additional ontology information; a decision circuit configured to determine whether the first ontology is to be merged with the additional ontology information based on external information including at least one of search preferences, user preference, social networking search engine data, web search engine data, or external ontologies; and a second ontology determination circuit configured to determine a second ontology based on the first ontology and the additional ontology information if it is determined that the first ontology is to be merged with the additional ontology information.

Brief Description of the Drawings

[0008] In the drawings, like reference characters generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various embodiments are described with reference to the following drawings, in which:

FIG. 1 illustrates a simplified diagram of an exemplary multimedia analysis framework of a method according to various embodiments;

FIG. 2 illustrates a simplified flow diagram of an exemplary process of controlling ontology based on a method according to various embodiments;

FIG. 3 illustrates a simplified block diagram of an exemplary system with which various embodiments may be used;

FIG. 4 illustrates a simplified block diagram of an exemplary system with which various embodiments may be used under limited available computation resources to operate in full capacity;

FIG. 5 shows a simplified block diagram of exemplary computing environment with which at least one multimedia storage, at least one search/retrieval unit, at least one metadata store may be implemented; and

FIG. 6 shows a simplified block diagram of computing environment which may be used for additional usage based on semantic multimedia analysis. Description

[0009] Embodiments described below in context of the devices are analogously valid for the respective methods, and vice versa. Furthermore, it will be understood that the embodiments described below may be combined, for example, a part of one embodiment may be combined with a part of another embodiment.

[0010] In this context, the ontology determination device as described in this description may include a memory which is for example used in the processing carried out in the ontology determination device. A memory used in the embodiments may be a volatile memory, for example a DRAM (Dynamic Random Access Memory) or a nonvolatile memory, for example a PROM (Programmable Read Only Memory), an EPROM (Erasable PROM), EEPROM (Electrically Erasable PROM), or a flash memory, e.g., a floating gate memory, a charge trapping memory, an MRAM (Magnetoresistive Random Access Memory) or a PCRAM (Phase Change Random Access Memory).

[0011] In an embodiment, a "circuit" may be understood as any kind of a logic implementing entity, which may be special purpose circuitry or a processor executing software stored in a memory, firmware, or any combination thereof. Thus, in an embodiment, a "circuit" may be a hard-wired logic circuit or a programmable logic circuit such as a programmable processor, e.g. a microprocessor (e.g. a Complex Instruction Set Computer (CISC) processor or a Reduced Instruction Set Computer (RISC) processor). A "circuit" may also be a processor executing software, e.g. any kind of computer program, e.g. a computer program using a virtual machine code such as e.g. Java. Any other kind of implementation of the respective functions which will be described in more detail below may also be understood as a "circuit" in accordance with an alternative embodiment.

[0012] According to various embodiments, devices and methods for semantic multimedia analysis may be provided.

[0013] The advances of information, communication and media technologies, has made available a huge amount of multimedia contents. Managing the varieties of multimedia contents such as video, audio, text documents and so on, have been a major challenge. When such multimedia contents are growing larger and larger, finding a desired set of multimedia or managing the overall operations around such multimedia store become difficult. In some cases, some of the multimedia contents may become invisible if not properly managed.

[0014] In order to make large multimedia stores searchable, adding metadata is a common practice. Such metadata may be added manually or automatically while automatic metadata generation is widely preferred due to its advantages such as minimum manual intervention, low cost and so on. This metadata generation involves manipulating low level features from multimedia (low level features such as visual features, auditory features or textual features and so on). Low level feature manipulation may further index certain concepts appears in multimedia contents (such as faces, locations and so on). A typical end user intends to search for a desired set of multimedia by giving a query. Such query may be complex in a practical situation. With the presence of a complex query (for example natural language like query), meta data manipulated from low level features may fail to provide an accurate set of search results or a desired set of multimedia contents to a given complex query. In order to solve semantic gap between metadata generated from low level features and complex or semantic queries knowledge representations such as ontologies are involved.

[0015] An ontology may provide a vocabulary that describes a domain of interest and a specification of the meaning of terms used in the vocabulary. Some of these vocabularies may identify from low level features and some may not due to the complex nature of certain multimedia (for example images and videos). Thus having ontology may reduce the semantic gap or semantic heterogeneity cause among low-level features based metadata and their associated meanings. Further, ontology like knowledge representation method may enhance metadata itself. For example in a case of sports ontology, when low-level feature based metadata includes; "ball, field, person, crowd and a like", such multimedia content may be further included or associated a high level metadata "football". Thus, such ontology based presentation also may use in new concepts detection (such as football) from low level features based metadata. On the other hand, when a complex query is given to such multimedia analysis system, query can be further mapped to ontology and reasoning on ontology may direct to the most relevant search results or desired set of multimedia. Thus, 1) supporting complex or natural language queries and 2) Concept detection indexing can be highlighted as the two main reasons for which semantic representation of multimedia needs to be considered. In order to make such ontology rich and advanced, fusion of multimodal data or features (visual, auditory and textual) is essential. Further, such multimedia analysis can be called "semantic multimedia analysis". Additional use cases or functions may include in a semantic multimedia analysis system based on system configurations and requirements. [0016] However, in semantic multimedia analysis system which is automatic to a large extent needs to consider generating and evolving one or more ontologies automatically as well. When such ontology is growing or evolving automatically, there may be additional issues arise. For example, when ontology is evolving larger and deeper, there may be additional intermediate nodes, incorrect associations. Such additional intermediate nodes, incorrect associations may cause in incorrect search results and various performance issues. When ontology is getting larger, reasoning on such ontology may become high complexity computational task. On the other hand, it is essential to keep such ontology in an optimized size in order prevent such performance issues and inaccurate results as much as possible. However, existing methods fail to address these issues in general.

[0017] In addition, a typical and contemporary semantic multimedia analysis system may have to deal with different organizations, requirements, users and so on. Some of these concerns may include the following three items:

[0018] 1. Diversity of contexts and Metadata: In certain industries or companies or organizations, there are multiple practices or standards are used in different departments and divisions. Further, these metadata required or preferred by different user groups may be largely different. For example, in broadcasting industry, metadata requirements may differ based on genre of multimedia contents (ex: news, wild life, drama and so on). Thus it may be vital to consider the difference of metadata that may be preferred by different types of multimedia categories and communities who benefit from semantic multimedia analysis (even within the same organization or system). [0019] 2. Metadata Standards: Since the metadata is a key resource in modern multimedia content management, metadata tenderization has been an important topic among the communities. Especially in broadcasting like domain, various metadata standards such as PBCore, LSCOM and so on. These metadata standards enables multiple benefits includes interoperability of metadata between diverse systems. Thus, it may be important to enable such metadata standards in a semantic multimedia analysis system.

[0020] 3. Impact of external sources: A modern multimedia analysis system may also incorporate or combined with external sources such as external ontologies provided by various organizations or publicly available (ex: Wordnet). These external sources may incorporate in rich ontology construction, query to ontology mapping and performance enhancement. Thus it may be important to consider these external sources in order to deliver a state of art semantic multimedia analysis system.

[0021] By considering aforementioned limitations of semantic multimedia analysis systems, and requirements demanding by contemporary multimedia analysis systems, according to various embodiments, a novel semantic multimedia analysis system may be provided. Various embodiments may provide better semantic analysis system with enhanced performance, accurate results. Further, according to various embodiments, methods and apparatuses may be provided to use such enhanced system in complex and practical settings.

[0022] According to various embodiments, appropriate ontology control may allow limiting ontology evolvement while keeping the ontology in an optimized size and structure. Further, such ontology may minimize the gap between query space and concepts which are being analyzed or for which metadata is generated. Thus, according to various embodiments, methods and apparatus may be provided for appropriate or preferred level of ontology control while keeping the ontology in desired size and structure while minimizing the gap between query space and concepts which are being analyzed or for which metadata is generated. Further ontology control according to various embodiments may be done by using prior knowledge about the concepts needs to be analyzed and allowing the evolvement on ontology accordingly. Such prior knowledge may extract from search preferences, search history, external resources such as metadata standards, social networking sites, web search engines and so on.

[0023] Various embodiments are directed to techniques and arrangements for processing and analysis of multimedia contents which may be in multiple formats or modalities such as video, audio, text, music and so on. In some cases, the multimedia analysis system needs to extract varieties of contexts and/or semantics embodied in multimedia contents which are being processed and analyzed. These contexts and/or appear in multimedia contents and their meanings are being commonly represented as one or more ontologies. Further, such ontologies may be automatically created and evolved by a computer program or computational algorithms. However, these automatically evolving ontologies require control, in order to prevent performance issues in a given system or framework. Thus, various embodiments as described herein may perform multimedia processing and analysis in order to create and control ontologies which are automatically created and automatically evolving. Moreover, implementations may be control and manage the ontologies dynamically to optimize overall performance of its functions. The key goals of such automatically evolving ontology would be to assist semantic analysis, semantic searching, and indexing/ metadata generation of multimedia contents.

[0024] Control and managing of ontology may be essential to avoid wrong search results and to enhance search performance. Various embodiments may further introduce methodologies to control and manage the automatically evolving ontology based on keyword preference. For example, automatically evolving ontology on each dedicated server/node will grow within a limited ontology scope which is derived using key word preferences or preferred set of keywords. This keyword may basically consist of textual keywords. However, such keywords may also include visual or auditory keywords as well. Further, methods and apparatus according to various embodiments may derive keyword preference from a given set of sample queries, search history, search engine and social network trends and so on.

[0025] For illustrative purposes, various embodiments are described in the environment of a computing device performing multimedia analysis. However, it will be understood that various embodiments are not limited to the particular examples provided, and may be extended to other types of devices, other execution environments, other system architectures, and so forth, as will be apparent to those of skill in the art in light of the disclosure herein.

[0026] According to various embodiments, metadata may be textual, numeric or multimedia or a combination thereof.

[0027] According to various embodiments, a query can also be a multimedia or combination of multimedia. For example, video or image or audio can also be given as a query. When a frequent keyword is a video or an image or any visual form, visual and textual ontologies may be merged to create or extract new concepts.

[0028] According to various embodiments, an ontology for an extracted concept may be changed automatically. In other words, in addition to automatically evolving, an ontology may automatically be updated or deleted accordingly as well.

[0029] FIG. 1 shows an illustration 10 of an example multimedia analysis framework which mainly aims for generation of semantic metadata and related operations including semantic multimedia analysis and searching according to various embodiments. For instance, the framework may receive plurality of multimedia contents 1 10 such as video, images, text, audio, music and so on. In addition to raw multimedia materials such as video, images, text, audio, music and so on, these material from multimedia 1 10 may also include preprocessed data from multimedia contents. Such preprocessed data from multimedia contents may result from operations such as on-screen character recognition, optical character recognition, and speech recognition and so on.

[0030] According to various embodiments, speech to text data may also take forms of logical descriptions. Such logic representations may for example include predicate logic, first order logic or alike. Further, the ontology relations may also take forms of aforementioned logical representations (for example first order logic). These logical representations may further be used for supporting semantic queries or ontology related operations described in the specification.

[0031] Multimedia 1 10 materials may further processed to extract their corresponding visual features 153, audio features 151 and textual features 152 (may include additional features such as geometric features and so on). These extracted features represent properties and nature of each modality (visual, auditory and textual) in form of a mathematical structure. These extracted features may further used in analysis of plurality of multimedia contents. These extracted features may further be stored in metadata store 140.

[0032] Visual feature extraction 153 may perform multiple operations related to visual information extraction. Visual feature extraction 153 may perform feature extraction considering regions within an image or a video frame, set of regions in multiple images or video frames and so on. In some cases, the region may be the area of a whole image or video frame as well. Further, the regions of interest may detect using dedicated computational algorithms. For example, regions of a human face may be selected followed by a face detection algorithm or module.

[0033] After visual, auditory and textual features are extracted and stored in metadata store 140, basic detectors 161 may further process and analyze the features stored in metadata store 140. Basic detectors 161, may consists of set of algorithms, models or computational programs or a like to detect and/recognize fundamental concepts appears in multimedia contents (such as person, pole, body, fire, etc). Examples of tasks performed by basic detectors 161 herein include: genre detection, identification or other recognition tasks: such as for faces, objects, locations, scenes, actions, logos, emotions, genders, costumes, furniture, and so on; context or genre recognition and identification (e.g., wild life, drama, news, etc.); optical character recognition; on-screen character recognition; gesture recognition; texture recognition, classification, and other texture- related operations; feature tracking; pose estimation and/or tracking, including position and orientation estimation of, e.g., a texture, a marker, a person, an object, a logo, a place, a location, and so forth; object tracking; motion estimation; motion tracking; motion segmentation; motion compensation; disparity computation; stereo rectification; image segmentation; and image registration. Further, while several examples of video/audio/text analysis tasks are described herein, other types of multimedia analysis tasks may be carried out depending on the requirements of a particular system.

[0034] Basic detectors 161 may further incorporate a separate multimedia corpus in order to assist the detection or recognition. For example, in order to index a particular person, the basic detectors 161 module may incorporate similar images or particular person's face. Further, in order to index or recognize a music clip, the basic detector may incorporate similar music clips stored in aforementioned multimedia corpus.

[0035] According to various embodiments, visual feature extraction 153, textual feature extraction 152, audio feature extraction 151, basic detectors 161 , storing extracted features and outcome of basic detectors 161 in metadata store 140 may be composed of a series of producer/consumer stages, each dependent on the output of a predecessor for perform a particular overall video analysis task of the respective pipeline.

[0036] According to various embodiments, visual feature extraction 153, textual feature extraction 152, audio feature extraction 151, basic detectors 161, storing extracted features and outcome of basic detectors 161 in metadata store 140 may be conducted using multiple servers or computing unit. Using of multiple servers may depend on system configuration and requirements.

[0037] According to various embodiments, one or more post processing modules may also perform various other post processing tasks which have not indicated in the FIG. l.Such tasks may include; physical multimedia segment generation, multimedia censorship management , image augmentation and so on.

[0038] Whilst the basic detectors 161 may consider features extracted from individual modality (i.e. visual, textual or auditory), the fusion module 162 may consider all available modalities (for example visual, auditory, and/ or textual) in a combined manner to derive certain information. For example, suppose there is no basic detector 161 available to detect "football scenes". According to various embodiments, the fusion module 162 may extract information or semantic related to a complex event or scene such as football. When an event like "football" scene, the fusion module may associate all visual, auditory and textual features related to "football". For example, when the word "football" appears in textual features, the fusion module may explore the related visual features extracted from 153 and basic visual concepts detected from 161. Such exploration may be done considering one or more multimedia contents whose visual, textual and auditory features are already extracted. In such cases, fusion module may determine the concept football may consists of "person", "crowd", "ball", "field" like atomic concepts detected by basic detectors 161. Further, the fusion module 162 may explore related auditory features extracted from 151 and basic audio concepts such as "cheering" detected from basic detectors 161 module. Thus, the fusion module may recognize the event of scene "football" may incorporate with "person", "crowd", "ball", "field" visual concepts and auditory concepts such as "cheering". This information extracted related to "football" scene like high level concept may further sent to new concept detection module 163. [0039] New concepts detection module 163 may receive information from the fusion module 162. For example in aforementioned "football" scene example, new concepts detection module 163 may receive information related multiple attempts which made my by fusion module 162 related to scene "football". By manipulating frequency, evidence or consistency of fusion module 162 outputs to a give concept such "football", the new concepts detection module 163 may create an ontology related to a given concept like "football". In such cases, football ontology may have properties such as person, ball, field, crowd detected from visual features, cheering like properties or concepts extracted from auditory features and additional textual features such as "sport", "ground", "yellow card", "red card", "time out " and so on.

[0040] Aforementioned ontology (for example "football ontology") extracted by new concepts detection module 163, may be used for multiple operations.

[0041] An ontology extracted by new concepts detection module 163, may be used to further enhance the indexing or metadata generation. For example, if a given video segment consists of concepts detected by basic detectors module 161 related to football ontology, such as person, field, ball, cheering, sports like properties in football ontology, the indexing and annotation module 170 may index such video segments with an identification of "football" scene. In such cases, the ontology extracted by new concept detection module 163 acts as a complex concepts detector which is based on the output of extracted features (151, 152, 153) and basic detectors 161.

[0042] On the other hand, an ontology extracted by new concepts detection module 163, may be used to further enhance the automatic evolving ontology 121 in learning module 120. This ontology which is automatically evolving in 120 may further be used to assist multiple operations such as semantic multimedia searching. The methods and apparatus discuss herein further relates to controlling and managing of this automatically evolving ontology 121, in order to have better performance and accuracy of a multimedia analysis system.

[0043] The framework illustrated in FIG. 1 may use external resources 100 for efficient and effective multimedia analysis and related tasks such as metadata generation, semantic multimedia search and so on. The external resource 100 may consists of different types of resources or feeds such as search preferences 101, social networking (SNS) or web search engine data 102, external ontologies 103 defined by various communities and various types of metadata standards 104. Selection of such external resources 100 may depend on requirements demand by a particular system.

[0044] Ontology control 130 module may further use the information from external resources 100 in order to control the ontology learning 120 and automatically evolving ontology 121. The ontology control 130 module may use the highly importance and atomic keywords in order to control or allow the growth of automatically evolving ontology 121. Moreover, these highly importance and atomic keywords are extracted from external resources 100. Extracting high important keywords or terms or concepts from external systems assumes to provide prior knowledge of key concepts or contexts or keywords related particular multimedia analysis and related operations.

[0045] FIG. 2 shows a flow diagram 200 illustrating an example Ontology Control 130 process according to various embodiments. Such ontology control 130 process may be used to control the automatically evolving ontology and its related operations. Controlling the automatically evolving ontology may cause in enhance performance (processing time, searching time and so on), enhancing accuracy and saving computational resources.

[0046] Multiple external resources 100 may be extracted by a keyword determination 131 module. Keyword determination module 131 may further derive highly important set of keywords, concepts and terms (for example; meeting, rain, protest and so on) from the external resources. Such highly important keywords are further sent to a restricted keyword ontology extraction 132 module. The main tasks of 132 would be to extract ontology related to high important keywords extracted from 131 in a restricted fashion.

[0047] As shown in FIG. 2, the restricted keyword ontology extraction module 132 may first search for the sentences where the important keywords appear (132 A). These sentences may have been previously extracted from a speech recognition process and stored in metadata store 140. If a particular keyword is found in some of the sentences extracted from speech recognition data or any related material, particular keyword's ontology is extracted from an externally available ontology based on a distance limit (132D and 132 F).

[0048] However, if such high important keyword derived from 131 (for example: meeting) is not available in speech recognition data, the system may further explore similar words or terms in a thesaurus. The process may continue to search associated keywords from the speech recognition data or alike. This associated keyword searching may also be incorporated similar terms extracted from an external thesaurus.

[0049] According to various embodiments, there may exist a minimum depth (levels of hierarchy) for a restricted ontology extracted for a given keyword or concept. In case where keyword ontology cannot be found or not found in an external ontology but the depth is not adequate enough, the process may further continue to find associated keywords to the given main keywords in sentences appears in speech to text data.

[0050] Extracting associated keywords to a given main keyword or similar term to a main keyword (for example "meeting") may consist of several stages. Such search for associated keywords 132B may first look for the sentences where the main keyword appears. Thereafter, sub keywords from the sentences may further be extracted based on a frequency based threshold. For example, if a particular word appears 2 or more times in sentences where main keyword (for example "meeting") appears, such keywords may further be shortlisted as "candidate associated keywords". The process may continue to calculate distance for each "candidate associated keywords" to the main keyword. Such distance measure would be the number of words in between main keyword and "candidate associated keywords" in each sentence. Further based on the within sentence distances and frequency of "candidate associated keywords" appearance, final set of "associated keywords" may be derived.

[0051] Once keywords and associated keywords are extracted in 132, a temporary ontology among keywords and associated keywords may be created randomly or by associating external ontology or information. According to various embodiments, the sentences derived in aforementioned example may also be used to construct this ontology by analyzing the grammar, vocabulary like language specific features. This ontology may further be identified herein as "keyword ontology" or "restricted keyword ontology".

[0052] Once an important keyword is extracted and its limited ontology is extracted, next step is to align a keyword's ontology with the main ontology which is automatically evolving. In order to make this alignment, keyword ontology and current state main ontology may be compared with external ontologies to find the closest element of keyword ontology to the main ontology (133 A). This comparison may further be conducted based on a distance measure (for example path based distance, path /depth based distance, and path/ information content based distance and so on), similarity measure or a like. Once such element is identified, the main ontology may be updated (133) with particular keyword ontology.

[0053] Once an important keyword is extracted and its limited ontology is extracted, restricted keyword ontology extraction module 132 may also notify the new concept detection module 163 and fusion module 162 in order to conduct multi -modality fusion and new concept detection for the elements related to a particular keyword's restricted ontology. The new concept detection module 163 and fusion module 162 may look for existing multimedia contents and/or incoming multimedia contents in order to detect and index new concepts related to the particular restricted keyword ontology.

[0054] Fusion module 162 may further look for same words or similar words in existing or incoming multimedia contents. Storing of concepts or ontology related to a particular search term or keyword may further store in storages 163B and 162 A which may be used by new concepts detection module 163 and fusion module 162 in order to analyze existing or incoming multimedia contents related to afore mentioned ontology.

[0055] Once a restricted keyword ontology extracted by 132 is further enriched and validated by new concept detection module 163, such ontology may further be added to the automatically growing ontology 121. If a similar ontology already exists in the main automatically growing ontology 121, it may also be updated accordingly. In order to make this alignment, the sub ontology enriched and validated by new concept detection module 163, and main ontology at the time 121 may be compared with external ontologies to find the closest element of keyword ontology to the main ontology (133 A). This comparison may further be conducted based on a distance measure (for example path based distance, path /depth based distance, and path/ information content based distance and so on), similarity measure or a like.

[0056] FIG. 3 shows an illustration 300 of an example system which may use the methodology according to various embodiments.

[0057] According to various embodiments, there may be different genre of multimedia which demands its own considerations. These considerations may include genre specific operations such as indexing, annotation, and clustering, semantic analysis and so on. The system illustrates in FIG. 3 may be used in aforementioned situations.

[0058] As illustrated in FIG. 3, the system consists of a main server 310 and multiple slave nodes dedicated for each genre news 381 , drama 382, and wild life 383 and so on. The genres specified in the system and the configuration of the system used, may vary based on the intended use of the system and the desired results to be achieved. Further, each of this slave nodes 381,382,383, 384, etc., may initiate with same or different ontologies. Such ontologies may arise from existing, manually defined metadata standards 380. In some cases, multiple slave nodes may be allocated to analyze contents related to the same genre. Such consideration may depend on amount of multimedia contents which are expected to be analyzed for a particular genre of multimedia.

[0059] These sub ontologies related to different genres of multimedia may further evolve automatically according to the methods and flows described corresponding to FIG. 1 and FIG. 2. [0060] However, the external resources used to extract keywords may be identical or different for each genre of multimedia. For example, search preference may differ from news genre to drama genre. In such cases ontology control 130 may be different in each slave nodes (i.e. 381 ,382,383,384, and so on). Further, in such cases external resources 100, may extract or analyzed on main server 310 and multiple slave nodes as well.

[0061] According to various embodiments, the system may extract plurality of multimedia contents 1 10 and extract visual features 153, textual features 152, and audio features 151 which are then followed by storing in metadata store 140. Further, multimedia segmentation 330 may be conducted based on multi modal (visual, auditory and textual) features extracted. Multimedia segmentation 330 may segment a video, audio, text or like based on contextual similarity or a similar process. For example, Multimedia segmentation 330 may segment a portion of a video where same person, location or scene appears.

[0062] The supervised genre classification 340 module may classify one or multiple segments of multimedia contents in to pre-identified genres such as news, wildlife, and drama and so on. The supervised genre classification 340 may consists of mathematical or machine learning models which may have been trained using example datasets provided. The mathematical models and/or machine learning models used in supervised genre classification 340 may further improved or modified dynamically.

[0063] Multimedia contents segmented by 330, may further be read by the supervised genre classification module 340 and may decide on one or may genres where a particular multimedia segment related to. In some cases, the supervised genre classification module 340 may classify the whole multimedia content in addition to classification on segments. [0064] Once the genre of a multimedia content or segments of a multimedia content is decided, that information may send to the related slave node or nodes (381, 382, 383, 384... etc.). Register job for relevant node 350, may direct the information such as name, id (identifier), timecodes, start and end signs of a multimedia content and so on the related slave node. Job registration may further include sending commands to relevant slave node indicated the need for analysis for a given multimedia content or segment. In cases where multiple slave nodes are allocated to analyze same genre of multimedia, 350 may also include balancing load or multimedia contents to be analyzed among those multiple servers.

[0065] Each slave node may conduct the methods and processes described corresponding to FIG. 1 and 2 in order to have a automatically evolving ontology which may unique to a particular genre of multimedia contents. The slave nodes may further conduct fusion 162 and new concepts detection 163 in order to analyze the related multimedia contents semantically. The results of semantic analysis may further send to metadata store 140 store the details such as indexes, metadata and so on.

[0066] Each slave node may also maintain its own metadata store depending on the system configuration and requirements. Further when a search request is given to the system, the distributed querying module 320 may search for relevant results incorporating the metadata store 140 and/or sub metadata stores maintained by each slave node such as 381,382,383 and so on.

[0067] Shared global ontology 360 may assist ontology control module 130, slave nodes 381,382,383 and distributed querying module 320. Shared global ontology 360 may consists of ontologies or related vocabularies, information extracted from external sources, websites, and standards and so on. According to various embodiments, shared ontology may also create from speech recognition or text or similar type of data/features without any restriction. Aforementioned ontology alignment and semantic query mapping may further benefit from the Shared global ontology 360.

[0068] Ontology control 130 may consists of in multiple instances. For example, each slave node may have its own sub Ontology control which is related to a particular genre of multimedia contents or user groups and divisions and so on.

[0069] Distributed querying module 320, may receive complex types of queries which intend to search multimedia contents or segments related to the given query. Such complex queries may consists of textual portion which are like natural language and may also include other multimedia materials such as images, video, audio and so on. In such cases, Distributed querying may decide appropriate slave nodes to be searched for the results. The textual portion of a query may be compared with respect to ontology of each slave node 381,382,383 and so on. Such comparison may lead to the identification relevant slave nodes where the query is possible related to. The querying module may further search metadata store 140 and or shared global ontology 360 to locate relevant genres related to a given query and retrieve multimedia contents or segments related to a particular query.

[0070] Distributed querying module 320 may receive complex queries from searching and retrieval module 180. Each time a new query is received by Distributed querying module 320, it may also update search preference 101 dynamically. Distributed querying module 320 may further retrieve relevant multimedia contents or segments related to a given query by incorporating resources or information from indexing and annotation module 170, automatically growing ontology 121, metadata store 140 and similar modules in slave nodes 381,382,383 and so on.

[0071] Service Bus 370 may allow communication between main server 120 and slave nodes 381, 382, 383 and so on. Service bus 370 may also allow adding new slave nodes or removing any existing slave node dynamically.

[0072] FIG. 4 shows an illustration 400 of an example system and apparatus where the system operates under limited available resources or the available resources are not fully enough to analyze multimedia contents in full capacity, according to various embodiments.

[0073] Various embodiments may be directed to techniques and arrangements for aforementioned embodiments under limited available resources or the available resources are not fully enough to analyze multimedia contents in full capacity. Thus, implementations herein may perform multimedia analysis and related operation on selected portions of multimedia rather than all multimedia contents. In some cases, when the available computational resources are limited for the system and functions to be operated in its full capacity, skipping certain amount of multimedia contents without analyzing is inevitable or such results after analysis may not be available on time. Thus, various embodiments may further be directed to techniques to prioritize multimedia portions which may analyze with a high priority while the other multimedia portions may be skipped or rescheduled to analyze when the system resources available.

[0074] Further, priority of a particular multimedia segment or a multimedia content may decide based on external resources 100 and ontology control 130. For example once multi modal features extracted 151, 152, 153, such features (for example textual features) may be compared or matched with the high important keywords appear in external resources 100, or restricted keyword ontologies 132. In addition, this comparison may also be done with respect to new concept detected in 163. If such high important keyword appears in a multimedia segment or whole multimedia content, such multimedia segment or multimedia content may categorize as a high priority multimedia content to be analyzed.

[0075] The priority described above may be further in cooperated in a system where computational resources may not be fully adequate. For example, when the implementations described in the figures operate under limited computer resources, resources of the slave nodes 381,382,383 which are dedicated to handle each genre of multimedia may continuously monitored. Once the genre classification 340 categorizes the slave node based on its genre, the resources of the relevant slave node may be inspected in 450. If the resource of the particular slave node is not adequate enough, the priority of the particular multimedia content may checked in 450. If such multimedia content contains high priority keywords or features, the system may further allocate resources to such slave node 430.

[0076] Prior to allocate more resources to a slave node in order to analyze a related multimedia content, the ongoing tasks may also be checked where low priority multimedia segments or whole multimedia contents may be analyzed 410. If such low priority tasks are being analyzed, such low priority tasks may be stopped 440 by the main server 310 by sending relevant commands to the particular slave node.

[0077] However, if there are no low priority tasks being processed or analyzed in a particular slave node which may possibly need to analyze a high priority multimedia contents, resources from other slave nodes may allocated to the particular slave node which is intended to analyze a high priority multimedia content 430.

[0078] Once the resource allocation or free enough resources is completed for a high priority multimedia content, information related to multimedia content and related details may register in the relevant slave node 350.

[0079] If no resources and multimedia low priority are present, such task may be may skipped temporally and rescheduled to process later when resources are available.

[0080] FIG. 5 shows a simplified diagram 50 of a illustrative computing system 500 in which semantic multimedia analysis according to various embodiments may be implemented. The computing system 500 may include multiple components, devices, interfaces and alike.

[0081] The illustrative computing unit 500 may communicate with one or more other computing, storage systems via network 580. Such computing and storage units may include one or more Metadata store 140, one or more searching and retrieval units 180. In addition, other computing or storage systems may also include external resources 100 or links to external resources 100 which may arise externally to the main computing devices 500. As described above, such external resources may include search preferences 101, dynamic information, data retrieval from social networking, web sites or web search engines 102, External ontologies 103 and metadata standards 104. Some of these external resources such as search preferences 101, external ontologies 103, metadata standards 104 may manually be added or configured or added/edited/modified through an application programming interface or may automatically be derived based on system features and configurations. [0082] According to various embodiments, it may be associated with one more multimedia storage systems 110. Such multimedia storage 110 may include storage devices HOB and storage interfaces 11 OA which allow communication with internal storage devices HOB and external computing systems, through a network 580.

[0083] According to various embodiments, multiple computing devices similar 500 may operate in a distributed fashion via a network 580. According to various embodiments, some of the components may appear in all the computing devices while some components may appear in certain devices only. For example, distributed querying module 320 may only appear in a computing device which may identified as the main computing device while other computing devices may not consist of distributed querying module 320. However, the core components related to analysis of multimedia may appear in all the computing devices or in majority of computing devices. These core components may include basic detectors 161. fusion 162, new concept detection 163, ontology learning 120, ontology control 130, feature extraction components 151,152,153 and so on.

[0084] According to various embodiments, feature extraction components 151, 152, and 153 may appear in separate set of computing devices in order to make those operations faster and efficient. Likewise, in order to make the operations and functions efficient and effective, components within computing devices 500 may be changed or modified accordingly.

[0085] There may be several other components such as cameras 570, OCR 116 (optical or on-screen character recognition), ASR 115 (automatic speech recognition) and alike. In some cases, these components may be provided in separate computing devices, where those components' outcome may be stored either in metadata store 140, or multimedia storage 1 10 depending on the nature of output given by those modules.

[0086] There may be additional components in computing devices 500 which have not been indicated in FIG. 5. Such additional components may be attached to post processing components such as censorship management, context based advertising, augmenting multimedia and so on.

[0087] Additional data or information storages or databases may also be attached to the components specified in FIG. 5. For example, one or more external relational database systems, or one or more unstructured data bases may be attached with searching and retrieval 180 and/or indexing and annotation 170 via the network 580, in order to provide more advanced searching and retrieval, indexing and annotation functions.

[0088] According to various embodiments, the computing devices 500 may further include or may be attached to external server components such as play out servers, web servers and so on.

[0089] According to various embodiments, all or part of multiple computing devices 500 may operate in a cloud infrastructure. In such cases, multimedia storage 110 and metadata 140 may also operate in cloud infrastructure depending on system configuration and requirements.

[0090] The methods, apparatus and systems according to various embodiments may be used basically in semantic multimedia analysis systems. Such semantic multimedia system may include primary functionalities such as indexing of multimedia contents and or indexing of segments within multimedia contents. Moreover, such system may allow providing complex queries to retrieve multimedia contents efficiently and effectively from a small scale or large scale multimedia store. In addition, such system may be used in specific type of multimedia content such as video. Further, such semantic multimedia analysis system may allow exporting metadata to internal or external systems for further use of these metadata. At secondary level, various embodiments may further be used to provide personalized search results based on individual or group of users' preference based on search preference of those individuals or groups.

[0091] In addition to aforementioned usage, semantic media analysis may be used for additional functionalities or services within the same or in an external system. For example, various embodiments may further be used or for following use cases:

[0092] - Multimodal search: Various embodiments may be used to search one or more modalities (for example visual, textual or auditory or alike) simultaneously by providing a complex query consists of multi modal inputs as well. Such multimodal query may include any combination of text, video clip, audio clips or alike.

[0093] - Context/Concept Search and indexing: Various embodiments may be used to search multimedia contents based on underlying concept or context. Thus, such operations may include further identification of scenes, event, elements or alike related to such contexts or concepts within multimedia contents.

[0094] - Contextual Content Monetization and Advertising: Aforementioned contextual analysis may further be used for advanced use cases such as content monetization and advertising. For example, once the concepts, contexts are extracted by advertisements may be directed to more targeted audiences with different interests and preference. The same methodology can be extended to content monetization as well. [0095] - Media rights and parental control: Various embodiments may further be incorporated in enforcing rights concerns in semantic multimedia analysis system. For example, by recognizing one particular multimedia which is owned by a specific entity can be checked over other multimedia over misuse. Further advanced, parental control and censorship management may also be possible with various embodiments. For example, multimedia may include (or consist of) contexts associated with such censorship related topics or vocabularies or scenes may easily be able to identified and act accordingly.

[0096] - Hierarchical Multimedia Clustering or segmentation: furthermore, various embodiments may be used to categorize or cluster multimedia as a whole or as individual modalities. For example, various embodiments may be incorporated in audio/music categorization (based on artist, composer, and instruments and so on), movies/videos segmentation (credits, scenes, breaks, commercials and so on),

[0097] FIG. 6 shows a simplified diagram 50 including an illustrative computing system 600 with additional components additional to the computing system illustrated in FIG. 5. Such additional components may be used in additional use cases such as contextual content monetization, censorship management, user preference analysis and so on.

[0098] The illustrative computing unit 600 may communicate with multiple applications such as administration 610 applications and end user application 690.

[0099] Administration 610 applications may include components to support administration, monitoring and controlling of computing devices, its components and its functionality. For example, administration 610 may include an interface to modify the automatically growing ontology 121, 121 's configurations and its related operations. Further, the administration 610 may also include an interface to visualize existing ontology through ontology visualization 612. Administration 610 may further support system management 613 which includes server, storage and computing devices management, database administration, system or end users management and so on. System management 613 may further include distributed servers 381, 382, 383, etc management. System management 613 may further include additional functions according to a particular system's configuration and requirements.

[00100] According to various embodiments, the computing system 600 may include several other components for further analysis of mining of information or knowledge based on automatically growing ontology 121. Such components may include search preference analysis 620, user preference analysis 630, suggestion analysis 640, and Sentiment analysis and so on.

[00101] Search preference analysis 620 may incorporate data and information from external resources 100 as well as other data stores such as user profiles 622, user profiles 622, search and retrieval history 623. However, according to various embodiments, search preference analysis 620 may incorporate data and information such as from sentiment data 626, suggestions data 625 and so on. Search preference analysis may further shortlist set of keywords and or multimedia contents which will further be used by ontology control 130. Once the search preference 621 short listed set of keywords and/or multimedia contents, those information may send to keyword determination module 131 for further steps. In order to shortlist keywords and/or multimedia contents, search preference may incorporate a mathematical modal which may analyze probability, statistics, machine learning or similar parameters of input data, Further, this mathematical model may use to rank possible keywords, multimedia contents with a higher productivity. All or some of the keywords (based on rank of keywords or multimedia contents), multimedia contents derived from search preference analysis 620, may send to ontology control 130 to conduct semantic multimedia analysis.

[00102] User preference analysis 630 module may further provide analysis of preferred search terms, multimedia contents, contexts for each individual user. In order to conduct user preference analysis 630, it may incorporate data from search preference 621, functions in search preference analysis 620, and particular user's search and retrieval history 623. User preference analysis 360 may further derive sub-ontologies from automatically growing ontology 121 and may store those data and ontology in user preferences data base 624. These sub-ontologies may further be used by distributed querying module 320 to derive most appropriate search results for a given user. In other words, user preference analysis module 630 may use to provide personalized search results.

[00103] Suggestion analysis 640 may deal with analyzing search preferences, individual user preferences as well as group-wise search preference in order to provide or suggests or recommends addition multimedia contents to a particular user or user group. The user groups may be identified based on user preference data 624. Suggestion analysis 640 may analyze users' profiles with mutual user preference sub ontologies. Frequency, time codes of mutual sub ontologies may further incorporate to decide upon a particular group. For example, within a particular time period, if two users' sub ontologies in user preference data 624 are above a predefined frequency threshold, those two may be categorized as logical group among the existing users pool. Once logical user groups are identified, the suggestions analysis 640 may calculate distance among all sub ontologies of all users in a particular group and may cluster all the sub ontologies. This sub ontology clustering may be done as a whole or by considering individual user. Further, for an individual user or for the whole user group, new multimedia suggestions may be given based on aforementioned clustering results. This clustering may finally derive scope of sub ontologies from automatically growing sub ontology 121 where a particular group of similar users are interest of.

[00104] In addition to sub ontologies, suggestion analysis 640 may also incorporate other information such as particular user's age, gender, race, country and alike details to decide on logical user groups. According to various embodiments, these data may be given by end users of the system which may have stored in user profiles data store 622. Suggestion analysis module 640 may further associate specific external resources (such as websites, social media, RSS feeds and so on) or external resources with specific parameters which may relate to an identified logical user group. Such external resources may further be used to derive new suggestions and recommendations. Once new recommendations or suggestions are extracted from suggestion analysis 640, those suggestions may be directed to ontology control 130 for further semantic multimedia analysis for the newly derived suggestions or recommendations.

[00105] Sentiment Analysis module 650 may analyze opinions, predictions, rankings or similar properties associated with all multimedia contents in multimedia storage 110. The sentiment analysis results may further be stored in a sentiment data 626. These sentiment analysis data may be used by applications such as end user application 690 for multiple purposes. Further sentiment analysis data 626 may further be used by external applications via an application programming interface or a like.

[00106] Sentiment analysis 650 may analyze information related to each individual multimedia content and/or segments of multimedia contents. Sentiment analysis 650 may analyze particular multimedia content's (whole or segments of a multimedia content) viewing frequency by end users, end users feedback in order to determine the popularity, opinion, likeability and so on. Sentiment analysis may analyze sentiment of sub- ontologies in automatically growing ontology 121 by using use user preference data 624, suggestions data 625. Sub ontologies appearing in user preference data 624, suggestions data 625 and their group statistics, individual users' feedback 187 may be used to analyze sub ontology sentiment. Further, the sub ontology sentiment may also be used to determine sentiment of associated multimedia contents' (associated with each sub ontology in) sentiment. This sentiment analysis may incorporate mathematical models, intelligent computational algorithms, text mining or alike to determine quantitative or qualitative sentiment values associated with sub ontologies or multimedia contents (whole or segments). According to various embodiments, the sentiment value may be determined using both sub ontology and multimedia sentiments. The sentiment analysis 650 may further predict, recommend, and suggest possible search terms (keywords and/or multimedia contents).

[00107] According to various embodiments, Censorship Management 660 component may be used to control censorship related material. In such cases, censorship related ontologies, sample multimedia contents, terms may store in a censorship data store 627. Further, censorship management may notify ontology control module 130, indexing and annotation module 170 to notify the censorship management module 660 regarding the multimedia contents where censorship sensitive materials or context take place. Further, censorship management may match multimedia contents (given to a particular user as search results) with censorship sensitive data stored in censorship data store 626 to make necessary actions such as augmenting multimedia contents, prohibiting a particular search result for users under a certain age and so on.

[00108] According to various embodiments, there may be advertisements attached with search results. The individual advertisers or advertising agencies or automated advertisement management agent may add advertisements to be shown with search results given to target set of users. Advertisements may be sent by an end user application 690 based on interest of the advertiser. A particular advertiser may see the sentiment data 626, and decide upon optimal set of contexts, concepts or multimedia contents that a particular advertiser is interested in. Further, advertiser may send the message or banner or a like the advertiser wish to appear for set of users via the push advertisement module. Such advertisement message may include video, image, audio, text, html or a like and such information may store in advertisement data store 628. Further, advertiser may use advertisement policy management 693, to configure number of advertisements, frequency of advertisements of a given user, targeted geographical area of an advertisement and so on. Inputs from 692 and 693 may further be stored in advertisement data store 628. End user application 690 may further include a payment module to make payments for the advertisements accordingly.

[00109] Advertisement Management 670 may decide advertisements to augment or include with search results through multimedia preparation 681. According to various embodiments, advertisement management 670 may allow multiple advertisers to bid for a given context or multimedia content or for a targeted users group. Advertisement management 670 may decide upon which multimedia should be augmented with whose advertisements and may send such details to multimedia preparation module 681 to prepare search results with advertisements accordingly.

[00110] Advertisement module may further store the details of advertisements sent to multimedia preparation, user's statistics and so on in advertisement data 628. These data may further be sent to a particular advertiser for reviewing purposes.

[00111] Furthermore, the uses of various embodiments may not be limited to above mentioned scenarios. Various embodiments may further incorporate embodiments where combined analysis of multimedia contents is vital. For example, various embodiments may also allow better enterprise search, educational resource management systems and so on.

[00112] Ontology based semantic representations are used widely in semantic multimedia analysis. An automatically evolving ontology is preferred by such semantic multimedia analysis systems in order to represent semantics or associated meanings, concepts of underlying multimedia. However, when such ontology is automatically evolving automatically, inaccurate associations may take place within elements of the ontology and these wrong associations may provide wrong search results. Typically, such ontology may also be associated with numerous other operations such as post processing operations which may also be may get affected due to incorrect associations in the ontology. Moreover, when ontology is growing and the size become larger and deeper, it may also contains large amount of unnecessary nodes which might not be used frequently or not using at all. Thus, querying on the ontology become high complexity task. Moreover, when such ontology is become unnecessarily larger, additional performance issues such as long multimedia searching time may take place. In addition, it may require or consume addition computational resources such as processing power, storage size and so on. Thus it is important to avoid in accurate results and keep the automatically growing ontology in a reasonable size to obtain optimal results while having an optimal system setting.

[00113] In order to avoid many issues arise from an automatically evolving ontology it is important to keep the ontology growing limited. However, the main challenge in keeping such ontology in an optimized way is related to what concepts needs to be indexed, annotated or recognized from a particular multimedia analysis system. If such essential concepts are known prior to the analysis process, there is a possibility to control and manage the automatic evolvement of an ontology. However, in practice it is difficult to decide upon such essential concepts for which metadata should be generated or analysis should be performed.

[00114] In order to solve the issues with an automatically evolving ontology and its related operations in a semantic multimedia analysis system, according to various embodiments, methods and apparatus may be provided for control and management of such ontologies. In order to make the decision upon which concepts needs to be analyzed or for which concepts metadata should be generated, various embodiments may incorporate prior knowledge of possible search terms or queries. The prior knowledge may be extracted from external resources such as metadata standards, social networking sites, web search engines and so on. Further, aforementioned prior knowledge may also include search preferences to a particular semantic multimedia analysis system. Such search preference may arise from users preferred search terms or search history or like. Aforementioned prior knowledge may further be used to determine on essential concepts needed to analyze. In such cases, concepts/ keywords and their associated concepts/keywords are extracted in a restricted manner. Further, the restricted amount of concepts needs to be analyzed and their associated concepts extracted in a restricted manner, are used to evolve ontology automatically in a restricted manner. Through these methods and apparatus, various embodiments may provide ontology control in order to avoid issues and limitations of commonly used methods.

[00115] Moreover, according to various embodiments, computational systems may be provided wherein such ontology control can be used effectively to solve the issues and limitations arise from commonly using methods. Thus, according to various embodiments, methods and systems may be provided for large scale semantic multimedia analysis while considering practical consideration as well as overall ontology control. Further, methods and systems may be provided to achieve optimal results for semantic multimedia analysis systems which may operate under limited computational resources. Further, various embodiments may provide faster and accurate semantic multimedia indexing, fast and most relevant multimedia searching. Further, various embodiments may provide significant reduction of computational resources needed for semantic multimedia analysis due to analysis under control of ontology. In addition, various embodiments may provide additional use cases based on semantic multimedia analysis. Various embodiments may enhance the performance and scope of such additional use cases such as content monetization, sentiment analysis, censorship management and so on.

[00116] According to various embodiments, an ontology determination method may be provided. The ontology determination method may include: determining a first ontology; determining additional ontology information; determining whether the first ontology is to be merged with the additional ontology information based on external information including at least one of search preferences, user preference, social networking search engine data, web search engine data, or external ontologies; and determining a second ontology based on the first ontology and the additional ontology information if it is determined that the first ontology is to be merged with the additional ontology information.

[00117] According to various embodiments, determining the second ontology may provide automatic evolving of the first ontology.

[00118] According to various embodiments, determining whether the first ontology is to be merged with the additional ontology information may include or may be or may be included in determining at least one keyword based on the external information.

[00119] According to various embodiments, determining whether the first ontology is to be merged with the additional ontology information may include or may be or may be included in extracting a restricted keyword ontology based on the determined at least one keyword.

[00120] According to various embodiments, determining whether the first ontology is to be merged with the additional ontology information may include or may be or may be included in determining a restricted ontology alignment based on the restricted keyword ontology.

[00121] According to various embodiments, determining whether the first ontology is to be merged with the additional ontology information may include or may be or may be included in extracting a feature from multimedia data. According to various embodiments, the feature can be a textual and/mathematical representation, a data structure and so on.

[00122] According to various embodiments, the multimedia data may include or may be or may be included in at least one of video, audio, text, music, or speech data.

[00123] According to various embodiments, the feature may include or may be or may be included in at least one of a visual feature, a textural feature, or an audio feature.

[00124] According to various embodiments, determining whether the first ontology is to be merged with the additional ontology information may include or may be or may be included in fusing the extracted feature with a restricted keyword ontology.

[00125] According to various embodiments, determining whether the first ontology is to be merged with the additional ontology information may include or may be or may be included in determining whether a new concept is included in the features based on the fusing.

[00126] According to various embodiments, it may be determined whether the first ontology is to be merged with the additional ontology information further based on a depth of the first ontology.

[00127] According to various embodiments, it may be determined whether the first ontology is to be merged with the additional ontology information further based on a depth of an ontology resulting from merging the first ontology with the additional ontology information.

[00128] According to various embodiments, determining the second ontology may include or may be or may be included in updating the first ontology based on the additional ontology information.

[00129] According to various embodiments, determining the second ontology may include or may be or may be included in appending the additional ontology information to the first ontology.

[00130] According to various embodiments, the ontology determination method may further include discarding the additional ontology information if it is determined that the first ontology is not to be merged with the additional ontology information.

[00131] According to various embodiments, the ontology determination method may further include keeping the first ontology unchanged if it is determined that the first ontology is not to be merged with the additional ontology information.

[00132] According to various embodiments, the ontology determination method may further include carrying out an application based on the second ontology.

[00133] According to various embodiments, the application may be censorship management, advertisement, sentiment analysis, and/ or user preference analysis.

[00134] According to various embodiments, the ontology determination method may further include providing advertisement based on the second ontology.

[00135] According to various embodiments, an ontology determination device may be provided. The ontology determination device may include: a first ontology determination circuit configured to determine a first ontology; an additional ontology information determination circuit configured to determine additional ontology information; a decision circuit configured to determine whether the first ontology is to be merged with the additional ontology information based on external information including at least one of search preferences, user preference, social networking search engine data, web search engine data, or external ontologies; and a second ontology determination circuit configured to determine a second ontology based on the first ontology and the additional ontology information if it is determined that the first ontology is to be merged with the additional ontology information.

[00136] According to various embodiments, determining the second ontology may provide automatic evolving of the first ontology.

[00137] According to various embodiments, the decision circuit may be configured to determine at least one keyword based on the external information.

[00138] According to various embodiments, the decision circuit may be configured to extract a restricted keyword ontology based on the determined at least one keyword.

[00139] According to various embodiments, the decision circuit may be configured to determine a restricted ontology alignment based on the restricted keyword ontology.

[00140] According to various embodiments, the decision circuit may be configured to extract a feature from multimedia data.

[00141] According to various embodiments, the multimedia data may include or may be or may be included in at least one of video, audio, text, music, or speech data.

[00142] According to various embodiments, the feature may include or may be or may be included in at least one of a visual feature, a textural feature, or an audio feature. [00143] According to various embodiments, the decision circuit may be configured to fuse the extracted feature with a restricted keyword ontology.

[00144] According to various embodiments, the decision circuit may be configured to determine whether a new concept is included in the features based on the fusing.

[00145] According to various embodiments, the decision circuit may be configured to determine whether the first ontology is to be merged with the additional ontology information further based on a depth of the first ontology.

[00146] According to various embodiments, the decision circuit may be configured to determine whether the first ontology is to be merged with the additional ontology information further based on a depth of an ontology resulting from merging the first ontology with the additional ontology information.

[00147] According to various embodiments, the second ontology determination circuit may be configured to update the first ontology based on the additional ontology information.

[00148] According to various embodiments, the second ontology determination circuit may be configured append the additional ontology information to the first ontology.

[00149] According to various embodiments, the decision circuit may be configured to discard the additional ontology information if it is determined that the first ontology is not to be merged with the additional ontology information.

[00150] According to various embodiments, the decision circuit may be configured to keep the first ontology unchanged if it is determined that the first ontology is not to be merged with the additional ontology information. [00151] According to various embodiments, the ontology determination device (for example an application circuit of the ontology determination device) may be configured to carry out an application based on the second ontology.

[00152] According to various embodiments, the application may be censorship management, advertisement, sentiment analysis, and/ or user preference analysis.

[00153] While the invention has been particularly shown and described with reference to specific embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. The scope of the invention is thus indicated by the appended claims and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced.

Claims

Claims What is claimed is:

1. An ontology determination method comprising:

determining a first ontology;

determining additional ontology information;

determining whether the first ontology is to be merged with the additional ontology information based on external information comprising at least one of search preferences, user preference, social networking search engine data, web search engine data, or external ontologies; and

determining a second ontology based on the first ontology and the additional ontology information if it is determined that the first ontology is to be merged with the additional ontology information.

2. The ontology determination method of claim 1 ,

wherein determining the second ontology provides automatic evolving of the first ontology.

3. The ontology determination method of claim 1 or 2,

wherein determining whether the first ontology is to be merged with the additional ontology information comprises determining at least one keyword based on the external information.

4. The ontology determination method of claim 3 ,

wherein determining whether the first ontology is to be merged with the additional ontology information comprises extracting a restricted keyword ontology based on the determined at least one keyword.

5. The ontology determination method of claim 4,

wherein determining whether the first ontology is to be merged with the additional ontology information comprises determining a restricted ontology alignment based on the restricted keyword ontology.

6. The ontology determination method of any one of claims 1 to 5,

wherein determining whether the first ontology is to be merged with the additional ontology information comprises extracting a feature from multimedia data.

7. The ontology determination method of claim 6,

wherein the multimedia data comprises at least one of video, audio, text, music, or speech data.

8. The ontology determination method of any one of claims 6 to 7,

wherein the feature comprises at least one of a visual feature, a textural feature, or an audio feature.

9. The ontology determination method of any one of claims 6 to 8, wherein determining whether the first ontology is to be merged with the additional ontology information comprises fusing the extracted feature with a restricted keyword ontology.

10. The ontology determination method of claim 9,

wherein determining whether the first ontology is to be merged with the additional ontology information comprises determining whether a new concept is included in the features based on the fusing.

11. The ontology determination method of any one of claims 1 to 10,

wherein it is determined whether the first ontology is to be merged with the additional ontology information further based on a depth of the first ontology.

12. The ontology determination method of any one of claims 1 to 11,

wherein it is determined whether the first ontology is to be merged with the additional ontology information further based on a depth of an ontology resulting from merging the first ontology with the additional ontology information.

13. The ontology determination method of any one of claims 1 to 12,

wherein determining the second ontology comprises updating the first ontology based on the additional ontology information.

14. The ontology determination method of any one of claims 1 to 13, wherein determining the second ontology comprises appending the additional ontology information to the first ontology.

15. The ontology determination method of any one of claims 1 to 14, further comprising:

discarding the additional ontology information if it is determined that the first ontology is not to be merged with the additional ontology information.

16. The ontology determination method of any one of claims 1 to 15, further comprising:

keeping the first ontology unchanged if it is determined that the first ontology is not to be merged with the additional ontology information.

17. The ontology determination method of any one of claims 1 to 16, further comprising:

carrying out an application based on the second ontology.

18. The ontology determination method of claim 17,

wherein the application comprises at least one of censorship management, advertisement, sentiment analysis, or user preference analysis. An ontology determination device comprising:

a first ontology determination circuit configured to determine a first ontology; an additional ontology information determination circuit configured to determine additional ontology information;

a decision circuit configured to determine whether the first ontology is to be merged with the additional ontology information based on external information comprising at least one of search preferences, user preference, social networking search engine data, web search engine data, or external ontologies; and a second ontology determination circuit configured to determine a second ontology based on the first ontology and the additional ontology information if it is determined that the first ontology is to be merged with the additional ontology information.