WO2017034309A1 - Method and apparatus for classifying media data - Google Patents

Abstract

Disclosed are a method and an apparatus for classifying media data. When receiving media data including still images or videos from a plurality of user terminals, the apparatus for classifying media data classifies the media data into space units on the basis of location information included in the media data, classifies the media data into time units on the basis of time information included in the media data, and classifies the media data into time-space units by combining the space units and the time units.

Description

Method and apparatus for classifying media data

The present invention relates to a method and apparatus for classifying media data, and more particularly, to a method and apparatus for automatically classifying a plurality of media data provided by a plurality of reporters for each event.

Recently, thanks to the development of smartphone camera technology, among the press pictures used in the media, photos taken by the smartphone are increasing rapidly. In particular, as the unexpected events such as traffic accidents and crime scenes, the proportion of video contents acquired by smartphones is gradually increasing. It is estimated that less than 5% of all photographs are reported for personal media such as smartphones and personal media, but less than 5% of all photographs are used for paper media or broadcast media. SNS media such as Wikitree and Shanghaist are estimated at over 15%.

In the case where the media and the like receive video reports from a large number of ordinary people, it is highly likely that the same events and reports of different personalities will be followed at the same time. Moreover, when a large-scale event occurs and there are a large number of reporters who are willing to report the video data, the media and the like will be provided with a large amount of video data in a short time.

However, when such a large amount of reports are made at the same time, it is practically impossible for the editors of the media to efficiently classify and interpret them to achieve the promptness of the report.

The present invention has been made in an effort to provide a method and apparatus for classifying a plurality of media data received from a plurality of reporters for each event.

In order to achieve the above technical problem, an example of a method of classifying media data according to the present invention includes: receiving media data including a still image or a video from a plurality of user terminals; Classifying the media data into space units based on location information included in the media data; Classifying the media data into units of time based on time information included in the media data; And classifying media data into space-time units by combining the space units and the time units.

In accordance with an aspect of the present invention, there is provided an apparatus for classifying media data, including: a receiver configured to receive media data including a still image or a video from a plurality of user terminals; A space unit classification unit classifying the media data into space units based on the location information included in the media data; A time unit classification unit classifying the media data into time units based on time information included in the media data; And a situation classifying unit classifying a situation provided by the media data based on keywords identified from sound or text information included in each media data classified by space and time.

According to the present invention, media data received from a plurality of user terminals may be classified in units of space time. In addition, based on voice or text information included in media data classified by space-time units, the media data can be automatically classified and provided, for example, a traffic accident or a fire. Without doing so, the media manager can easily grasp the type and importance of the incident.

In addition, media data is categorized into three dimensions based on the space where the event occurred, so that the administrator can easily view the event from the point of view desired. In addition, the size of the space unit and time unit for classifying the media data can be actively changed according to the place where the event occurred (for example, urban center or camping ground) or the type of event (for example, traffic accident or fire). Accurately collect and classify media data for the incident.

1 shows a schematic structure of an entire system for media data classification according to the present invention;

2 is a flowchart illustrating an example of a method of classifying media data according to the present invention;

3 illustrates an example of media data according to the present invention;

4 is a flowchart illustrating an example of a method of classifying media data into space units according to the present invention;

5 is a diagram illustrating an example of a method of classifying spatial units of media data according to the present invention;

6 is a diagram illustrating an example of changing a basic space unit according to the present invention;

7 is a flowchart illustrating an example of a method of classifying media data according to an embodiment of the present invention in units of time;

8 illustrates an example of a time unit classification method of media data according to the present invention;

9 illustrates an example of changing a basic time unit according to the present invention;

FIG. 10 is a diagram illustrating an example of classifying media data into space-time units according to the present invention; FIG.

11 illustrates an example of a method of determining a relative photographing position based on a photographing direction of media data according to the present invention;

12 illustrates an example of a method of determining a relative photographing distance based on a subject size of media data according to the present invention;

FIG. 13 illustrates an example of a method of determining a relative photographing height based on a photographing angle of media data according to the present invention; FIG.

14 is a view showing an example of displaying the recording position of the media data in three dimensions according to the present invention;

15 is a diagram illustrating an example of a method of classifying media data according to an indoor and outdoor reference according to the present invention;

16 is a view showing the configuration of an embodiment of a user terminal according to the present invention;

17 is a diagram showing the configuration of an embodiment of a media data classification apparatus according to the present invention.

Hereinafter, a method and apparatus for classifying media data according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating a schematic structure of an entire system for media data classification according to the present invention.

Referring to FIG. 1, the media data classification apparatus 100 for automatically classifying media data including a still image or a moving image receives media data from a plurality of user terminals 110, 112, 114, and 116 through a wired or wireless communication network.

The user terminals 110, 112, 114, and 116 refer to all types of terminals capable of transmitting media data to the media data classification apparatus through a wired or wireless communication network. An example of the user terminals 110, 112, 114, and 116 may be a smartphone or a tablet PC. An example configuration of the user terminals 110, 112, 114, and 116 will be described again with reference to FIG. 16.

The media data classification apparatus 100 classifies and provides media data received from the plurality of user terminals 110, 112, 114, and 116 in space-time units. As another example, the media data classification apparatus 100 may classify and provide media data according to an event type, importance, and the like. As another example, the media data classification apparatus 100 provides a three-dimensional relative photographing position of the media data so that the administrator can view the incident from a desired perspective.

2 is a flowchart illustrating an example of a method of classifying media data according to the present invention.

Referring to FIG. 2, the media data classification apparatus (hereinafter referred to as an “device”) receives media data from a plurality of user terminals (S200). The device is based on location information (for example, GPS (Global Positioning System) coordinate information, etc.) and time information (which may include date information according to an embodiment) included in the media data. The media data of the spatial region and the time zone in which various incidents occurred is classified (S210).

The device not only classifies the media data into spatial units (eg, two-dimensional or three-dimensional classification) and time units, but also according to the embodiment, the location where the media data is taken or the location of the event scene belongs to indoor or outdoor. Can be further classified on the basis of An example of a method of classifying media data in space units is illustrated in FIG. 4, and an example of a method of classifying media data in units of time is illustrated in FIG. 7. In addition, an example of a method of further classifying media data based on indoor and outdoor is illustrated in FIG. 15.

If the media data includes additional information such as recording information or text information, the device may further classify the media data on a case-by-case basis based on the additional information (S220).

Specifically, in the case where the media data includes recording information recording the event scene sound or the whistleblower's description voice, or text information on the scene situation, the device may be a variety of conventional speech recognition algorithms or text recognition algorithms. You can analyze the recorded data or text information to determine the type of event. For example, if the word information such as 'fire', 'fire truck', or 'fire' is high in the additional information included in the plurality of media data, the device identifies the event indicated by the media data as 'fire'. To this end, the device may have a word list for each event in advance.

The device may provide the classified media data to another device or display it on the screen (S230). For example, when the device receives a request for providing media data in a specific time zone of a specific region, the device classifies and provides the media data. In this case, the device provides a relative position of each media data according to an embodiment so that an administrator or the like can easily identify the incident scene represented by the media data from a desired perspective. As another example, the device may provide media data on a case-by-case basis, such as a fire or a traffic accident.

3 is a diagram illustrating an example of media data according to the present invention.

Referring to FIG. 3, the media data 300 transmitted from the user terminal to the device includes a still image (or video) 310, tagging data 320, and additional data 330.

The still image or video 310 may be an image photographed through a camera, or may be an image photographed directly by a camera of the user terminal or photographed by a device other than the user terminal. Hereinafter, for convenience of description, the still image or the video 310 will be described as limited to the case where the user terminal is photographed.

The tagging data 320 is data representing various environmental information at the time of image capturing, and includes position information indicating a photographing position and time information indicating a capturing time. In addition, the tagging data 320 may further include various information according to an embodiment, such as illuminance, humidity, temperature, a tilt of the user terminal, and a shooting direction.

The additional data 330 is information added by the informant, and includes recording information or text information. For example, the whistleblower may record a description of an event scene situation or record a scene sound at the time of shooting through a recording function of a user terminal, or may record a description of the event scene. The additional data 330 may further include various types of information added by the informant.

4 is a flowchart illustrating an example of a method of classifying media data into space units according to the present invention.

Referring to FIG. 4, the device determines the spatial distribution density of the media data based on the location information of the media data received from the plurality of user terminals (S400). In general, since a large number of reports will be made in case of an accident such as a fire or a traffic accident, the device detects the distribution density of media data as an example of a method for automatically identifying where an event occurred.

For example, referring to FIG. 5, the device determines the distribution density of each region by the basic space units 510, 512, and 514 having a predetermined area, and then identifies the region 510 that is greater than or equal to a predetermined threshold. However, even if the scene of the event is in the middle of the city and in a secluded rural village, the number of people who film and report the same kind of event may be very different. Therefore, the device may vary the threshold value of the distribution density for the classification of spatial units according to geographic information of each region (for example, large cities, small cities, urban centers and neighborhoods, residential areas, and malls).

When the device selects the space to be classified based on the distribution density, the device grasps the distribution center of media data of the corresponding area (S410). For example, there may be cases where the informant is skewed to one side of the space depending on the scene of the event, so that the device grasps and classifies the media data located in a certain space from the distribution center of the media data for more accurate media data classification. .

However, depending on the geographic environment where the event occurred or the type of event, the distribution range of the informant may vary. For example, in the case of a large forest fire, the informant can range from several kilometers, and in the case of a traffic accident, the informant can range from several meters away from the scene of the incident. Also, in the middle of a crowded city center, the area of the scene is narrow because the view is blocked by buildings or people, and in the case of a secluded countryside, the scene can be taken from a distance.

Therefore, the device does not apply the size of the basic spatial unit 510 of FIG. 5 to classify the media data as it is, but instead of the geographic information or the event type (the type of the central subject of the media data) where the event occurs as shown in FIG. 6. Actively change accordingly. The large circle 520 of FIG. 5 is an example in which the size of the basic space unit is changed. As illustrated in FIG. 2, the apparatus may determine the type of event based on the additional information when the media data includes additional information, or the central subject (ie, the image center) present in the still image (or video) of the media data. The type of event can be determined according to the type of the subject). For example, when the central subject of the media data is a car, the event type may be classified as a traffic accident, and when it is a flame or smoke, it may be classified as a fire event.

When the size of the space unit for classifying the media data is set (S420), the device grasps and classifies the media data located within the space unit set at the distribution center of the media data (S430). For example, in FIG. 5, the number of media data located in the right basic space unit 514 is four, but when the size of the space unit is increased (520), the number of media data is nine.

5 is a diagram illustrating an example of a method of classifying space units of media data according to the present invention, and FIG. 6 is a diagram illustrating an example of changing a basic space unit according to the present invention.

Referring to FIG. 5 together, the apparatus determines the distribution density of each media data based on the location information included in the media data. In this case, the device may determine the distribution of media data accumulated for a predetermined time. For example, the device may determine the distribution density of the media data received on an hourly basis.

Referring to FIG. 6, the apparatus changes the size of the basic space unit 600 according to the geographic information and the event type, and then grasps and classifies the media data located in the space units 610 and 620 of the changed size. For example, in the case of the first area (or the first event), if the range of where the informant is located is narrow, the device changes the basic space unit 600 into a small space unit 610. On the contrary, in the case of the second region (or the second event), if the reporter is located in a wide range, the apparatus changes the basic space unit 600 to a larger space unit 620. The device may also apply different thresholds for classifying spatial units according to geographic information or event type.

7 is a flowchart illustrating an example of a method of classifying media data according to an embodiment of the present invention in units of time.

Referring to FIG. 7, the device determines a time distribution density of media data based on time information of media data received from a plurality of user terminals (S400). In general, the report will increase in frequency when an event occurs, so the device uses the time distribution density of the media data as an example of a method for automatically identifying and classifying the occurrence time of the event.

As shown in FIG. 4, the number of informants may vary according to the geographic information or the type of the event where the event occurs. Therefore, the device may vary the threshold value of the time distribution density for time unit classification according to the geographic information where the event occurs and the type of event. For example, referring to FIG. 8, a time zone for event classification may be determined by using the A threshold value in the case of an incident site in a downtown area and a B threshold value in a rural area.

The apparatus grasps the time distribution center of the media data (S410), and identifies and classifies the media data in a predetermined time region based on the time distribution center. Since the time distribution of the report can be different depending on the geographic information or the type of event, the device actively changes the size of the time range (ie, time unit) for media data classification. For example, if an event lasts for a long time, such as a fire, it may be necessary to increase the size of time units for media data classification.

When the size of the time unit is set (S420), the device grasps and classifies the media data located within the set time unit at the center of time distribution of the media data (S430).

8 is a diagram illustrating an example of a time unit classification method of media data according to the present invention, and FIG. 9 is a diagram illustrating an example of changing a basic time unit according to the present invention.

8 and 9, the reporting time and the reporting amount of the informant may vary according to the location of the event or the type of the event. Accordingly, the device actively varies and applies the size of the threshold values A and B for classifying the time zone where an event occurs and the time units 910 and 920 for classifying media data according to geographic information or event type.

For example, when the threshold A is applied because the event location is downtown, the media data located within the time unit 1 when the threshold occurrence frequency 800 exceeds the threshold A in the event occurrence frequency 800 and 810 is identified and classified.

FIG. 10 illustrates an example of classifying media data in units of space time according to the present invention.

Referring to FIG. 10, the device may determine a space unit for classifying media data through the method of FIG. 4, and may determine a time unit for classifying media data through the method of FIG. 7. According to an embodiment, two methods of FIGS. 4 and 7 may be simultaneously applied or only one of the two methods may be applied.

The device classifies the media data located in the area 1000 by combining the space unit and the time unit. Therefore, the device can automatically identify the region and time zone where a particular event occurred and classify and provide media data about the event.

11 to 14 illustrate an example of a method of three-dimensionally displaying media data classified in units of space and time. 11 illustrates an example of a method of determining a relative photographing position based on a recording direction of media data according to the present invention, and FIG. 12 illustrates a relative photographing distance based on a subject size of media data according to the present invention. FIG. 13 is a diagram illustrating an example of a determination method, and FIG. 13 is a diagram illustrating an example of a method of determining a relative photographing height based on a photographing angle of media data according to the present invention.

Referring to FIG. 11, when the photographing direction information is included in the media data, the apparatus grasps the location information of the media data so that the photographing direction (arrow direction in FIG. 11) of the media data converges to one point 1150. The relative photographing positions 1112, 1122, 1132, and 1142 are determined by moving the photographing positions 1110, 1120, 1130, and 1140.

Referring to FIG. 12, the device determines the relative photographing distance (l ₁ , l ₂ , l ₃ , l ₄ ) between the photographing positions of the media data from the center subject based on the size of the center subject included in the media data. Determine the relative photographing positions (1200, 1210, 1220, 1230). This embodiment shows an example in which FIG. 12 is again applied to the result of FIG. 11, but an application order thereof may be changed in various ways according to the embodiment.

Referring to FIG. 13, if the tilt data is included in the media data, the device may determine a photographing angle of each media data. As the photographing angle is larger, the photographing heights 1300 and 1310 of the media data are higher, and the device determines the relative photographing heights h ₁ and h ₂ of the media data based on the photographing angles of the media data.

Referring to FIG. 14, the apparatus determines the three-dimensional position 1400 of the media data based on the relative photographing direction, relative photographing distance, and relative photographing height of each media data determined in FIGS. 11 to 13. Since the device determines and provides the 3D location 1400 of each media data based on the center of the event, the administrator can easily identify the scene of the event in a desired direction or height. According to an embodiment, the device may provide a screen on which media data is three-dimensionally arranged.

15 is a diagram illustrating an example of a method of classifying media data based on indoors and outdoorss according to the present invention.

Referring to FIG. 15, the device determines whether the photographing position of the media data is indoors or outdoors (S1500). For example, when the device cannot receive the GPS information of the user terminal when receiving the media data from the user terminal, or when the location information included in the media data and the recording time of the still image or video included in the media data When the information is different or when the location information is missing in the media data, the photographing position of the media data can be regarded as indoor.

The device also distinguishes whether the event site is also indoors or outdoors (S1510). For example, the device may distinguish whether the event scene is indoor or outdoor based on the spectrum analysis of the subject of the media data.

The device may classify the media data into four cases as described below based on the case where the recording location and the event site location of the media data are indoors and outdoors, respectively (S1520).

The scene of the event indoor The scene of the incident outdoors Shooting location is indoor Case 1 Case 2 Shooting location is outdoor Case 3 Case 4

16 is a diagram illustrating a configuration of an embodiment of a user terminal according to the present invention.

Referring to FIG. 16, the user terminal 110 includes a photographing unit 1600, a sensing unit 1610, a recording unit 1620, an input unit 1630, and a transmission unit 1640. In addition, the user terminal may include various types of sensors according to embodiments, such as the position sensor 1612, the time sensor 1614, the tilt sensor 1616, the direction sensor 1618, and the like.

The photographing unit 1600 captures a still image or a video of the scene of the incident through a camera.

The sensing unit 1610 measures and stores various surrounding conditions through a sensor when the photographing unit 1600 is photographed. For example, the sensing unit 1610 measures and stores location information and time information at the time of shooting through the position sensor 1612 and the time sensor 1614. If there is a tilt sensor 1616 such as a gyroscope in the user terminal, the sensing unit 1610 measures and stores the tilt of the terminal at the time of shooting, and the direction sensor 1618 that can grasp the direction of the terminal like the geomagnetic sensor is provided. If present, the sensing unit 1610 measures and stores a photographing direction when photographing.

The recording unit 1620 receives and records the sound of the scene of the incident or the voice of the whistleblower. The input unit 1630 receives and stores information such as text.

The transmitting unit 1640 may include tagging data including a still image (or a moving image) captured by the photographing unit 1600, various sensing information captured by the sensing unit 1610, and a recording unit 1620 or an input unit 1630. Media data including the additional data consisting of the recording information or text information received through the) is generated and transmitted to the media data classification apparatus 100.

17 is a diagram showing the configuration of an embodiment of a media data classification apparatus according to the present invention.

Referring to FIG. 17, the media data classification apparatus 100 includes a receiver 1700, a spatiotemporal classifier 1710, and a provider 1770. Specifically, the space-time classification unit 1710 may include a space unit classification unit 1720, a time unit classification unit 1730, a situation classification unit 1740, an indoor / outdoor classification unit 1750, and a three-dimensional classification unit 1760. have.

The receiver 1700 receives a plurality of media data from a plurality of user terminals.

The space-time classification unit 1710 classifies the plurality of media data by space and time at which an event occurs.

In detail, the space unit classification unit 1720 identifies a space in which an event occurs and identifies and classifies media data in the space. For example, the spatial unit classification unit 1720 may use the method illustrated in FIG. 4.

The time unit classifying unit 1730 identifies the time when an event occurs and identifies and classifies corresponding time zone media data. For example, the time classifier 1730 may use the method illustrated in FIG. 7.

When the additional data is included in the media data, the situation classifier 1740 identifies and classifies the types of events provided by the media data using the additional data.

The indoor / outdoor classification unit 1750 classifies the recording location of the media data or the location of the occurrence location of the media data according to whether the location is indoor or outdoor. For example, the indoor / outdoor classification unit 1750 may use the method illustrated in FIG. 15.

The three-dimensional classifier 1760 determines and provides a relative three-dimensional position with respect to a photographing position of each media data when the media data includes tilt information or direction information.

If there is a request for media data of a specific place or time zone, the provider 1770 classifies and provides the media data. Alternatively, the providing unit 1770 may identify an important event and automatically provide it to an administrator when the salping space distribution density or the time distribution density exceed a predetermined value in FIGS. 4 and 7, that is, when the number of reports is large. have. Alternatively, when an event type is determined, the provider 1770 may automatically classify and provide media data according to a predetermined importance for each event type.

The invention can also be embodied as computer readable code on a computer readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disks, optical data storage devices, and the like. The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

So far I looked at the center of the preferred embodiment for the present invention. Those skilled in the art will appreciate that the present invention can be implemented in a modified form without departing from the essential features of the present invention. Therefore, the disclosed embodiments should be considered in descriptive sense only and not for purposes of limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope will be construed as being included in the present invention.

Claims (11)

Receiving media data including a still image or a moving image from a plurality of user terminals; Classifying the media data into space units based on location information included in the media data; Classifying the media data into units of time based on time information included in the media data; And And classifying the media data into space-time units by combining the space units and the time units. The method of claim 1, And classifying an event type provided by the media data based on a frequency of a word included in voice or text information included in each media data classified by the space-time unit. 2. The method of claim 1, wherein the classifying of the space unit comprises: Identifying a spatial region in which the spatial distribution density of the media data is equal to or larger than a preset threshold value; Determining a spatial distribution center of media data in the spatial region; Setting the size of the space unit according to the geographic information where the location information of the media data is indicated or the type of the central subject of the still image or the moving image included in the media data; And And identifying media data located in a spatial area corresponding to the size of the set spatial unit at the spatial distribution center. The method of claim 1, wherein the classifying of the time unit comprises: Identifying a time domain in which the time distribution density of the media data is equal to or greater than a preset threshold value; Determining a center of time distribution of media data in the time domain; Setting the size of the time unit according to the geographic information where the time information of the media data indicates or the type of the central subject of the still image or the video included in the media data; And And identifying media data located in a time domain corresponding to the size of the set time unit at the time distribution center. The method of claim 1, Determining a photographing direction of the media data based on the direction information included in the media data; And And determining a relative photographing position of each media data such that the photographing directions of the respective media data all converge to one point. The method of claim 1, Determining a photographing angle of the media data based on the inclination information included in the media data; And And determining the relative photographing height of each media data based on the photographing angle of each media data. The method of claim 1, Determining a relative size of a central subject of a still image or a moving image included in each media data; And And determining a relative photographing distance of each media data from the central subject based on the relative size. The method of claim 1, Whether the recording location of the media data is indoors or outdoors depending on whether the creation time of the location information included in the media data and the recording time information of the still image or video included in the media data are the same or whether the location information is missing. Distinguishing; Discriminating whether a location of the subject is indoors or outdoors based on a spectrum analysis of a subject of a still image or a moving image included in media data; And And classifying media data based on whether the photographing location and the location of the subject are indoors or outdoors. A receiver configured to receive media data including a still image or a video from a plurality of user terminals; A space unit classification unit classifying the media data into space units based on the location information included in the media data; A time unit classification unit classifying the media data into time units based on time information included in the media data; And And a situation classification unit classifying a situation provided by the media data based on keywords identified from sound or text information included in each media data classified in a space unit and a time unit. The method of claim 9, wherein the space unit classification unit, The relative position of the recording position of each media data is grasped based on the direction information included in the media data, and the relative position of the recording position of each media data from the subject based on the size of the subject of the still image or the moving image included in the media data. And determining a three-dimensional position of each media data by determining a distance and determining a relative height of each media data from the subject based on the degree of inclination included in the media data. A computer-readable recording medium having recorded thereon a program for performing the method of claim 1.

Images