[go: up one dir, main page]

WO2020130309A1 - Dispositif de masquage d'image et procédé de masquage d'image - Google Patents

Dispositif de masquage d'image et procédé de masquage d'image Download PDF

Info

Publication number
WO2020130309A1
WO2020130309A1 PCT/KR2019/013173 KR2019013173W WO2020130309A1 WO 2020130309 A1 WO2020130309 A1 WO 2020130309A1 KR 2019013173 W KR2019013173 W KR 2019013173W WO 2020130309 A1 WO2020130309 A1 WO 2020130309A1
Authority
WO
WIPO (PCT)
Prior art keywords
masking
identification information
feature vector
candidate
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/KR2019/013173
Other languages
English (en)
Korean (ko)
Inventor
강지홍
임비
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Lomin Inc
Original Assignee
Lomin Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Lomin Inc filed Critical Lomin Inc
Publication of WO2020130309A1 publication Critical patent/WO2020130309A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
    • H04N21/4318—Generation of visual interfaces for content selection or interaction; Content or additional data rendering by altering the content in the rendering process, e.g. blanking, blurring or masking an image region
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion

Definitions

  • the present application relates to a video masking device and a video masking method, and relates to a video masking device and a video masking method capable of masking a video for personal privacy.
  • the conventional de-identification technology has a difficulty in that the user has to perform the de-identification processing for the corresponding video, particularly, in the case of de-identifying only a specific person among various persons included in the video There were difficulties such as having to manually check for all the characters included in the whole video.
  • the registration number 10-1215948 "a video information masking method of a surveillance system based on body information and face recognition" has been proposed.
  • the face recognition of a person is possible from the image information, the distance between the tail and the middle of the face is detected, and based on this, the face of a specific person stored in the database is identified, and the masking process of the person is separately processed. Is being presented.
  • This application is intended to provide a video masking device and a video masking method capable of masking a video to protect the privacy of an individual.
  • the present application is intended to provide an image masking apparatus and a method for masking images that can selectively mask objects included in a video.
  • the present application is intended to provide an image masking apparatus and an image masking method capable of efficiently masking by tracking objects included in a video.
  • the present application is intended to provide an image masking apparatus and a method for masking an image having an excellent detection rate and recognition rate in detecting, tracking, and recognizing an object included in a video.
  • An image masking method includes: generating a frame image in a frame unit by decoding an input target video; Extracting a frame feature vector corresponding to each frame image from the frame image, and detecting a candidate object from the frame feature vector; Extracting an object feature vector corresponding to each candidate object from the frame feature vector using the detected location information of the candidate objects; Setting identification information for the candidate objects using the object feature vector; Tracking candidate objects having the same identification information in a continuous frame image to generate tracking information for the candidate objects for each identification information; And when receiving the masking input, extracting candidate objects identical to the identification information included in the masking input using the tracking information and performing masking on the extracted candidate objects.
  • the setting of the identification information may include: searching a registration feature vector corresponding to the object feature vector in the identification information database; When a registration feature vector corresponding to the object feature vector is searched, extracting identification information matching the registration feature vector from the identification information database and setting the identification information of the candidate object; And if the registration feature vector corresponding to the object feature vector is not searched, generating identification information of the candidate object, and registering the object feature vector and the identification information in the identification information database.
  • the step of searching in the identification information database if the object feature vector matches the registered feature vector within a predetermined error range, it may be determined that the object feature vector corresponds to the registered feature vector.
  • the generating of the tracking information may track the same candidate object by using a difference value of an object feature vector of each candidate object included in the continuous frame image and a change in the position and size of the candidate objects. have.
  • the object characteristic vector of the first candidate object V2 is the object characteristic vector of the second candidate object included in the second frame image
  • d1 is the distance from the reference point to the center point of the first candidate object
  • d2 is the second candidate object's distance from the reference point.
  • the distance to the center point, s1 is the area of the first candidate object, s2 is the area of the second candidate object, a, b is the weight, and the second candidate object having the minimum tracking error value is the same as the first candidate object It can be determined as a candidate object having information.
  • an error is determined as an error in setting identification information for the candidate object It may further include a detection step.
  • the tracking information may include at least one of identification information of the candidate object, frame information where the candidate object appears, and location information and size information of the candidate object included in the frame image.
  • the image masking method using the tracking information, generating an object tracking image by overlaying the candidate object and the identification information for each candidate object on the target video It may further include.
  • a list of identification information arranged with identification information corresponding to candidate objects included in the target video, and candidate objects having the same identification information appear in the target video
  • the method may further include displaying a masking selection interface including appearance section information and a frame image in which the candidate object appears.
  • the masking input may be generated by using identification information input from a user or by extracting the identification information according to a preset selection algorithm.
  • the masking area may be masked by using blurring, mosaic processing, or image replacement.
  • the step of performing the masking may store masking information for a masking area set in the target video and the target video, and mask the target video using the masking information when playing the target video and play the target video. have.
  • the step of performing the masking may require an access right when the target video is played, and if the access right is not available, the target video may be masked and played.
  • a computer program stored in a medium may exist in combination with hardware to execute the image masking method described above.
  • An image masking apparatus for generating a frame image of a plurality of frames by decoding (decoding) the input target video; A frame feature vector corresponding to each frame image is extracted from the frame image, a candidate object is detected from the frame feature vector, and using the location information of the detected candidate objects, each candidate object from the frame feature vector A feature vector extraction unit for extracting an object feature vector corresponding to the; An identification information setting unit configured to set identification information for the candidate objects using the object feature vector; A tracking information generation unit that tracks candidate objects having the same identification information in a continuous frame image and generates tracking information for the candidate objects for each identification information; And a masking unit that, upon receiving the masking input, extracts candidate objects identical to the identification information included in the masking input using the tracking information, and performs masking on the extracted candidate objects.
  • An image masking apparatus the processor; And a memory coupled to the processor, wherein the memory includes one or more modules configured to be executed by the processor, and the one or more modules decode a received target video and perform frame-by-frame.
  • a frame image is generated, a frame feature vector corresponding to each frame image is extracted from the frame image, a candidate object is detected from the frame feature vector, and location information of the detected candidate objects is used to obtain the frame feature
  • the object feature vector is used to set identification information for the candidate objects, and tracking candidate objects having the same identification information in a continuous frame image
  • each object included in a video can be distinguished, a user-selected object can be selectively masked to be de-identified. That is, rather than collectively de-identifying the objects included in the video, it is possible to selectively de-identify the selected specific object, thereby improving user convenience.
  • the image masking apparatus and the image masking method according to an embodiment of the present invention since the position of each object included in the video can be tracked, a specific object selected by the user can be easily extracted and masked over the entire video have.
  • the image masking apparatus and the image masking method according to an embodiment of the present invention since it is possible to perform detection, tracking and recognition of a candidate object from one frame feature vector, it is efficient and machine learning of each detection, tracking and recognition Algorithms and the like can be used to learn to perform at the same time, which can significantly improve performance and computational speed.
  • the image masking apparatus and the image masking method according to an embodiment of the present invention by providing a masked video when an unspecified person who does not have access permission to view a video, it is possible to prevent personal information exposure and privacy infringement.
  • FIG. 1 is a schematic diagram showing an image masking system according to an embodiment of the present invention.
  • FIGS. 2 and 3 are block diagrams showing an image masking apparatus according to an embodiment of the present invention.
  • FIG. 4 is a schematic diagram showing candidate object extraction and identification information setting according to an embodiment of the present invention.
  • FIG. 5 is a schematic diagram showing a masking selection interface according to an embodiment of the present invention.
  • FIG. 6 is a flowchart illustrating an image masking method according to an embodiment of the present invention.
  • modules and “parts” for the components used in the following description are given or mixed only considering the ease of writing the specification, and do not have meanings or roles distinguished from each other in themselves. That is, the term'unit' used in the present invention means a hardware component such as software, FPGA or ASIC, and'unit' performs certain roles. However,'wealth' is not limited to software or hardware.
  • The'unit' may be configured to be in an addressable storage medium or may be configured to reproduce one or more processors.
  • 'part' refers to components such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, Includes subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, database, data structures, tables, arrays and variables.
  • the functionality provided within components and'parts' may be combined into a smaller number of components and'parts' or further separated into additional components and'parts'.
  • FIG. 1 is a schematic diagram showing an image masking system according to an embodiment of the present invention.
  • an image masking system may include an image photographing apparatus 1 and an image masking apparatus 100.
  • the image photographing device 1 may photograph a surrounding environment to generate a video.
  • the video photographing device 1 may be any device that can photograph a video, such as a video camera or a camcorder.
  • the video photographing device 1 may stream the photographed video in real time or store it as a file, and the video photographing device 1 may provide wired or wireless communication to transmit the photographed video.
  • the image masking apparatus 100 may receive a target video to be masked, and may perform masking on objects included in the received target video.
  • the image masking device 100 may receive the target video shot by the video recording device 1 in a file or data format, and in accordance with an embodiment, receives the target video streaming in real time from the video recording device 1 It is also possible.
  • the image masking apparatus 100 may include an image processing apparatus 1 such as a CCTV, a vehicle black box, or a camera. ), or may be provided in a separate computer or smartphone.
  • an image processing apparatus 1 such as a CCTV, a vehicle black box, or a camera.
  • the image masking apparatus 100 may distinguish objects included in the target video, and may perform masking on the distinguished objects. That is, in order to prevent exposure of personal information or invasion of privacy, some objects included in the video may be masked and de-identified. For example, in the case of a video recording device 1 such as a CCTV or a black box for a vehicle, since the set shooting area is randomly recorded, the face or body part of another person, a vehicle license plate, etc. may be photographed.
  • a video is released through the Internet or the like, personal information such as another person's face may be exposed, causing problems such as invasion of privacy. Therefore, in order to prevent such a problem, it is necessary to process such that it cannot be identified by masking the face or the license plate of another person included in the target video.
  • the image masking apparatus 100 it is possible to identify each object appearing in the video, and automatically performs masking by tracking only selected objects among objects included in the video It is possible to do.
  • FIG. 2 is a block diagram showing an image masking apparatus according to an embodiment of the present invention.
  • the image masking apparatus 100 includes a frame input unit 110, a feature vector extraction unit 120, an identification information setting unit 130, and tracking information generation unit 140. , An error detection unit 150, a masking unit 160, an object tracking image generation unit 170, and a masking selection interface display unit 180.
  • the frame input unit 110 may generate an image in a frame unit by decoding an input target video.
  • the frame input unit 110 may receive a target video encoded in a Moving Picture Experts Group (MPEC) method from the image photographing device 1, and then decodes the target video in the form of a compressed MPEC file, and the target video
  • MPEC Moving Picture Experts Group
  • Each of the frames of can be captured to generate a frame image.
  • the target video may include N frame images, and frame numbers from 1 to N may be assigned to each frame image.
  • N corresponds to a natural number.
  • the feature vector extracting unit 120 may extract a frame feature vector corresponding to each frame image from the frame image.
  • the frame feature vector generated by the feature vector extracting unit 120 may be used to detect candidate objects included in the frame image or to distinguish and recognize each candidate object, and is included in consecutive frame images. It can also be used to track objects.
  • the feature vector extracting unit 120 may generate a frame feature vector using pixel information of the frame image, such as location information and pixel value information of each pixel included in the frame image.
  • a machine learning algorithm such as a convolutional neural network (CNN) or a recurrent neural network (RNN) is used, or a histogram of oriented gradient (HOG) , LBP (Local Binary Pattern) can be used to extract the statistical characteristics of the image.
  • the frame feature vector can be generated in various ways, and the content of the present invention is not limited by the above-described method.
  • the feature vector extracting unit 120 may detect candidate objects included in each frame image by using the generated frame feature vectors.
  • the candidate object is an object that can be the object of masking, and the candidate object can be set differently according to embodiments.
  • CCTV Click Circuit Television
  • an object requiring masking may be a person or a vehicle license plate, etc.
  • the feature vector extraction unit 120 extracts a person or vehicle license plate, etc. from a target image captured by CCTV and sets it as a candidate object.
  • features such as shape and luminance of candidate objects to be extracted can be learned in advance using a machine learning algorithm, and the feature vector extraction unit 120 utilizes them to correspond to candidate objects from each frame image. Areas can be detected. For example, it is possible to repeatedly learn the shapes of various people and extract the shape of a person included in a frame image as a candidate object.
  • CNN, RNN, Principal Component Analysis (PCA), Logistic as a machine learning algorithm Regression, Decision Tree, etc. can be used.
  • the feature vector extracting unit 120 may specify and display the extracted candidate object as a bounding box or a segmentation mask on the frame image.
  • the bounding box may be displayed as a rectangle around the pedestrian, which is a candidate object, as shown in FIG. 4(a), and the user can easily distinguish candidate objects through the bounding box.
  • the position coordinates (x,y) of the upper left corner of the bounding box and four numbers (x,y,w,h) indicating the width and height (w,h) of the bounding box are used. .
  • Each bounding box can be specified.
  • a foreground corresponding to a candidate object among frame images may be displayed separately from the background, which is the remaining background, in pixels. That is, as illustrated in FIG. 4(b), the pixel values of pixels corresponding to the background may be set to 0, and the pixel values of candidate objects corresponding to the foreground may be set to 1.
  • the feature vector extraction unit 120 may generate an object feature vector corresponding to the candidate object.
  • the feature vector extracting unit 120 may specify a region corresponding to a candidate object among frame feature vectors, and extract a feature vector value corresponding to the specified region and set it as an object feature vector.
  • candidate objects can be distinguished using the object feature vector. For example, when the same candidate object A appears consecutively on a plurality of frame images, the object feature vectors of the same candidate object A may be set identical or very similar on each frame image. On the other hand, in case of different candidate objects, the object feature vector has an error value at least equal to or greater than a set value. Accordingly, candidate objects having the same object feature vector may be determined to correspond to the same object, and candidate objects having different object feature vectors may be determined to correspond to different objects.
  • the size of the candidate object may be changed for each frame image according to the position or moving direction of the candidate object, and accordingly, the size of the object feature vector may be set differently for each frame image. Therefore, the feature vector extracting unit 120 may uniformly transform the size of each object feature vector to a predetermined size using interpolation.
  • the feature vector extracting unit 120 may detect a candidate object and generate an object feature vector of a candidate object using the frame feature vector. That is, since candidate object detection and object feature vector generation can be performed using the extracted frame feature vector, efficient operation is possible and the operation speed can be improved.
  • the identification information setting unit 130 may set identification information for candidate objects using the object feature vector. That is, the identification information setting unit 130 may distinguish each candidate object by using the object feature vector of the candidate objects, and may display identification information for each distinguished candidate object.
  • FIG. 4(a) includes a plurality of candidate objects, and each candidate object corresponds to a different object. Accordingly, the identification information setting unit 130 distinguishes each candidate object and can set “ID” as “138”, “147”, “128”, and “153” as identification information for the distinguished candidate object, respectively. have. In this case, the identification information setting unit 130 may set identification information for each candidate object using an identification information database.
  • the identification information setting unit 130 may search the identification information database d for a registration feature vector corresponding to the object feature vector of candidate objects extracted from each frame image.
  • the identification information database (d) the registration feature vector and identification information corresponding to each registration feature vector may be matched and stored. Accordingly, the identification information setting unit 130 may search for identification information corresponding to the object feature vector of the candidate object in the identification information database d.
  • the identification information setting unit 130 may newly generate identification information corresponding to the object feature vector, and register the newly generated identification information and the object feature vector in the identification information database d to identify the identification information database (d ).
  • the identification information setting unit 130 may determine that the object feature vector corresponds to the registered feature vector when the object feature vector matches the registered feature vector within a preset error range. That is, even in the case of the same candidate object, since the object feature vector may include some errors for each frame image, it is possible to determine the identity in consideration of the error range.
  • the identification information setting unit 130 sets identification information of each candidate object with reference to the identification information database d, candidate objects having the same object feature vector among candidate objects included in the target video have the same identification information. Can be set. For example, if the candidate object having the object feature vector A appears in the frame image 3-10 and then again appears in the frame image 20-26, the candidate object is frame images 3-10 and 20-26 It may be set to have the same identification information b in the frame image.
  • the identification information database d may be provided for each target video, but according to an embodiment, the identification mask database 100 includes one identification information database d for all target videos. It is also possible. In this case, if the object feature vector included in the newly provided target video is the same as the object feature vector included in the previous target video, it may be set to the same identification information as the identification information set in the previous video. That is, the same identification information may be set for the same candidate object for different target videos.
  • the tracking information generating unit 140 may track candidate objects having the same identification information in a continuous frame image, and generate tracking information for candidate objects for each identification information. That is, the tracking information can be generated by combining the location information and the identification information of the candidate object extracted from each frame image, and the tracking information can be used to track the positional change of the candidate object that naturally follows in the continuous frame image. have.
  • the tracking information may include identification information of the candidate object, frame information such as the frame number where the candidate object appears, location information and size information of the candidate object included in the frame image. Further, depending on the embodiment, it is also possible to further include information on the pose of the candidate object, facial feature points, clothing feature points, and the like in the tracking information.
  • a single target video may include a plurality of tracking information, and each tracking information may be generated for each candidate object identification information.
  • the tracking information generator 140 tracks the same candidate object by using the difference value of the object feature vector of each candidate object included in the continuous frame image and the position and size change of the candidate objects. It is also possible to do. For example, when tracking a first candidate object included in a first frame image, tracking a plurality of candidate objects included in a second frame image subsequent to the first frame image, with the first candidate object Error values can be calculated. Thereafter, the candidate object having the smallest tracking error value may be determined as the same candidate object as the first candidate object.
  • Error is the tracking error value
  • V1 is the object characteristic vector of the first candidate object included in the first frame image
  • V2 is the object characteristic vector of the second candidate object included in the second frame image
  • d1 is the first from the reference point.
  • d2 is the distance from the reference point to the center point of the second candidate object
  • s1 is the area of the first candidate object
  • s2 is the area of the second candidate object
  • a and b are weights and can be assigned to any constant. It corresponds.
  • the candidate object can be tracked in a continuous frame image using the tracking error value described above.
  • V1-V2 is greater than or equal to a set limit error, it is possible to determine the second candidate object as a different candidate object without calculating the tracking error value. That is, since the object feature vector is outside the set error range, the second candidate object may be determined to be different from the first candidate object.
  • the error detection unit 150 may detect an error in setting identification information for a candidate object. For example, when a candidate object having the same identification information is partially missing in a continuous frame image, the error detection unit 150 may determine that an error has occurred in setting identification information for the candidate object.
  • a first frame image at a time t-1, a second frame image at a time t, and a third frame image at a time t+1 exist, and (x1 of the first frame image and the third frame image , y1)
  • the error detection unit 150 may notify the user or the like by displaying the occurrence of the error.
  • the masking unit 160 may extract candidate objects identical to the identification information included in the masking input using the tracking information, and perform masking on the extracted candidate objects. That is, the masking unit 160 may selectively mask the selected candidate objects, and the candidate objects to be masked may be identified using identification information.
  • the masking input may be generated using identification information input from a user, or may be generated using identification information extracted according to a preset selection algorithm.
  • the image masking apparatus 100 may provide a separate masking selection interface to the user, and the user may select identification information corresponding to a candidate object to be masked using the masking selection interface.
  • the masking unit 160 may receive the masking input through the masking selection interface, and may mask candidate objects corresponding to the received masking input.
  • candidate objects to be masked are automatically extracted using a separate selection algorithm set in the image masking apparatus 100, all candidate objects included in a specific section in the target video are identified to be masked. Information may be extracted, or identification information corresponding to a specific candidate object included in a target video may be extracted and masked. For example, using a selection algorithm, identification information corresponding to a specific gender or age group may be extracted from candidate objects included in the target video.
  • the masking unit 160 may extract the frame images in which the candidate object corresponding to the selected identification information appears using the tracking information, and set a masking area in correspondence with the position of the candidate object indicated in the extracted frame image. have. Thereafter, the set masking area may be masked to de-identify so that users cannot identify it.
  • the masking area may be limited to a region corresponding to a face among candidate objects.
  • the masking unit 160 may mask the masking area by blurring or mosaicing.
  • blurring may be implemented using a low-pass filter, and depending on the embodiment, masking may also be performed using image substitution that covers the masking area with a single color or a specific pattern, a separate image or animation, or a character.
  • the masking unit 160 may encode an image synthesized by masking, and allow a masked image to be output through an image output unit (not shown). Meanwhile, according to an embodiment, the masking unit 160 may separately store masking information and a target video for a masking area set in the target video. That is, after storing the original file of the target video separately, the target video may be masked using masking information when the target video is played.
  • the access right to the original file may be requested.
  • masking of the target video may be applied and provided.
  • the access right can be checked through various types of authentication such as password, fingerprint recognition, and iris recognition.
  • the image masking apparatus 100 may allow a user to select candidate objects to be masked, and in this case, may further include components for enhancing the user's convenience of selection.
  • the object tracking image generator 170 may generate an object tracking image by overlaying a candidate object and identification information for each candidate object on the target video using the tracking information. That is, as shown in FIG. 4(a), an object tracking image may be generated by displaying a boundary box representing the candidate object and identification information together with the target video.
  • the user can check the object tracking image and identify identification information of the candidate object being tracked and each candidate object. Accordingly, the user may select an object to perform a masking process among a plurality of candidate objects by referring to the object tracking image.
  • the masking selection interface display unit 160 may provide a masking selection interface so that a user can select candidate objects to be masked.
  • the masking selection interface can be implemented as shown in FIG. 5. That is, the masking selection interface may display a list of identification information in which identification information corresponding to candidate objects included in the target video is sorted, and the appearance interval information and the frame in which the candidate objects having the same identification information appear in the target video The number of images and the like can be displayed.
  • the user may further include a frame image in which the candidate object appears, so that the user can identify candidate objects corresponding to each identification information.
  • the user can select candidate objects to be masked in the target video through the masking selection interface, and can select candidate objects to be masked by checking the checkbox displayed next to the identification information list. That is, it is possible to selectively set candidate objects for masking, rather than collectively masking all objects included in the target video.
  • the image masking apparatus 100 may include a physical configuration, such as a processor 10, memory 40, memory 40
  • a physical configuration such as a processor 10, memory 40, memory 40
  • One or more modules configured to be executed by the processor 10 may be included therein.
  • the one or more modules may include a frame input module, feature vector extraction module, identification information setting module, tracking information generation module, error detection module, masking module, object tracking image generation module, and masking selection interface display module. .
  • the processor 10 may perform various functions and execute data processing functions by executing various software programs and a set of instructions stored in the memory 40.
  • the peripheral interface unit 30 may connect input/output peripheral devices of the image masking device 100 to the processor 10 and the memory 40, and the memory controller 20 may include the processor 10 or the image masking device 100. When a component of the access to the memory 40, it can perform a function for controlling the memory access.
  • the processor 10, the memory controller 20 and the peripheral interface unit 30 may be implemented on a single chip or may be implemented as separate chips.
  • the memory 40 may include high-speed random access memory, one or more magnetic disk storage devices, and non-volatile memory such as a flash memory device.
  • the memory 40 may further include a storage device located away from the processor 10 or a network attached storage device accessed through a communication network such as the Internet.
  • the image masking apparatus 100 the memory 40, the operating system, the frame input module corresponding to the application program, feature vector extraction module, identification It may include an information setting module, a tracking information generation module, an error detection module, a masking module, an object tracking image generation module, and a masking selection interface display module.
  • each module is a set of instructions for performing the above-described functions, and may be stored in the memory 40.
  • the processor 10 may access the memory 40 and execute instructions corresponding to each module.
  • the frame input module, feature vector extraction module, identification information setting module, tracking information generation module, error detection module, masking module, object tracking image generation module, and masking selection interface display module include the frame input unit, feature vector extraction unit, The identification information setting unit, the tracking information generation unit, the error detection unit, the masking unit, the object tracking image generation unit, and the masking selection interface display unit respectively correspond to each, so a detailed description thereof will be omitted here.
  • FIG. 6 is a flowchart illustrating an image masking method according to an embodiment of the present invention.
  • the image masking method may include a tracking information generation step (S50) and a masking step (S60).
  • each step may be performed by an image masking apparatus according to an embodiment of the present invention.
  • the received target video may be decoded to generate a frame-by-frame image.
  • the video masking device may receive an MPEC-encoded target video from the video photographing device, and in this case, decode a target video in the form of a compressed MPEC file, and then capture each frame of the target video to generate a frame image. Can.
  • the image masking apparatus may generate a frame feature vector using pixel information of the frame image, such as location information and pixel value information of each pixel included in the frame image, and according to an embodiment, CNN (Convolutional Neural) Machine learning algorithms such as Network), Recurrent Neural Network (RNN), and methods for extracting statistical characteristics of images such as Histogram of Oriented Gradient (HOG) and Local Binary Pattern (LBP) can be used.
  • CNN Convolutional Neural
  • RNN Recurrent Neural Network
  • HOG Histogram of Oriented Gradient
  • LBP Local Binary Pattern
  • the image masking apparatus may detect candidate objects included in each frame image using the generated frame feature vector.
  • the candidate object is an object that can be subjected to masking, and features such as shape and luminance of the candidate objects can be learned in advance using a machine learning algorithm. That is, CNN, RNN, PCA (Principal Component Analysis), Logistic Regression, Decision Tree, etc. can be learned using machine learning algorithms, and by using this, a region corresponding to a candidate object can be detected from each frame image. .
  • the extracted candidate objects may be specified and displayed on a frame image with a bounding box or a segmentation mask.
  • the object feature vector corresponding to each candidate object may be extracted from the frame feature vector using the detected location information of the candidate objects. That is, a region corresponding to a candidate object may be specified among frame feature vectors, and a feature vector value corresponding to the specified region may be extracted and set as an object feature vector.
  • candidate objects can be distinguished using the object feature vector.
  • the size of the candidate object may be changed for each frame image according to the position or moving direction of the candidate object, and accordingly, the size of the object feature vector may be set differently for each frame image. Therefore, the size of each object feature vector can be constantly modified to a predetermined size by using interpolation.
  • identification information for candidate objects may be set using an object feature vector. That is, each candidate object may be distinguished using the object feature vector of the candidate objects, and identification information may be assigned to each distinguished candidate object and displayed. At this time, identification information for each candidate object may be set using an identification information database.
  • a registration feature vector corresponding to the object feature vector of candidate objects extracted from each frame image may be searched in the identification information database.
  • the identification information database a registered feature vector and identification information corresponding to each registered feature vector may be matched and stored. Therefore, it is possible to search identification information corresponding to the object feature vector of the candidate object in the identification information database.
  • identification information matched with the registered feature vector can be extracted from the identification information database and set as identification information of the corresponding candidate object.
  • identification information corresponding to the object feature vector can be newly generated, and the identification information database can be updated by registering the newly generated identification information and the object feature vector in the identification information database.
  • the object feature vector matches the registered feature vector within a predetermined error range, it can be determined that the object feature vector corresponds to the registered feature vector. That is, even in the case of the same candidate object, since the object feature vector may include some errors for each frame image, it is possible to determine the identity in consideration of the error range.
  • candidate objects having the same identification information may be tracked in a continuous frame image, and tracking information for candidate objects may be generated for each identification information. That is, the tracking information can be generated by combining the location information and the identification information of the candidate object extracted from each frame image, and the tracking information can be used to track the positional change of the candidate object that naturally follows in the continuous frame image. have.
  • the tracking information may include identification information of the candidate object, frame information such as the frame number where the candidate object appears, location information and size information of the candidate object included in the frame image. Further, depending on the embodiment, it is also possible to further include information on the pose of the candidate object, facial feature points, clothing feature points, and the like in the tracking information.
  • a single target video may include a plurality of tracking information, and each tracking information may be generated for each candidate object identification information.
  • Error is the tracking error value
  • V1 is the object characteristic vector of the first candidate object included in the first frame image
  • V2 is the object characteristic vector of the second candidate object included in the second frame image
  • d1 is the first from the reference point.
  • the distance from the reference point to the center point of the candidate object, d2 is the distance from the reference point to the center point of the second candidate object, s1 is the area of the first candidate object, s2 is the area of the second candidate object, and a and b are weights and can be assigned to any constant. It corresponds.
  • an error detection step may be further included. That is, in the error detection step, an error in setting identification information for a candidate object may be detected, and when an error in setting identification information is detected, an error may be notified to the user. For example, there may be a case where a candidate object having the same identification information is partially missing within a continuous frame image, and in this case, an error in setting identification information for the candidate object may occur because the interval between each frame image is very short. It can be determined that it has occurred.
  • the masking step (S60 ) when the masking input is received, candidate objects identical to the identification information included in the masking input may be extracted using the tracking information, and masking of the extracted candidate objects may be performed. That is, the selected candidate objects can be selectively masked, and candidate objects to be masked can be specified using identification information.
  • the masking input may be generated using identification information input from a user, or may be generated using identification information extracted according to a preset selection algorithm.
  • the masking step (S60 ) frame images in which candidate objects corresponding to the selected identification information appear using the tracking information may be extracted. Subsequently, a masking area may be set in correspondence to the position of the candidate object indicated in the extracted frame image, and the set masking area may be masked to be de-identified so that users cannot identify it.
  • the masking area may be limited to a region corresponding to a face among candidate objects.
  • the image masking device may mask the masking area by blurring (blurring) or mosaicing.
  • blurring can be implemented using a low-pass filter.
  • the masking area may be masked by using a single color or a specific pattern, image substitution covering a separate image, animation, character, or the like.
  • the image masking device may encode the synthesized image of the masking, and may output the masked image using an image output unit.
  • the image masking device may encode the synthesized image of the masking, and may output the masked image using an image output unit.
  • the access right to the original file may be requested.
  • masking of the target video may be applied and provided.
  • the access right can be checked through various types of authentication such as password, fingerprint recognition, and iris recognition.
  • the image masking method according to an embodiment of the present invention may allow a user to select candidate objects to be masked, and in this case, may further include components for increasing the convenience of user selection.
  • an object tracking image generation step (not shown) may be further included, and by using the tracking information, the candidate object and identification information for each candidate object may be overlaid on the target video to generate the object tracking image. That is, an object tracking image may be generated by displaying a boundary box indicating the candidate object and identification information together with the target video.
  • the user can check the object tracking image and identify identification information of the candidate object being tracked and each candidate object. Accordingly, the user may select an object to perform a masking process among a plurality of candidate objects by referring to the object tracking image.
  • a masking selection interface display step may be further included to provide a masking selection interface so that a user can select candidate objects to be masked.
  • the masking selection interface may display a list of identification information in which identification information corresponding to candidate objects included in the target video is sorted, and appearance interval information in which candidate objects having the same identification information appear in the target video. The number of appeared frame images may be displayed.
  • an image of a frame image in which the candidate object appears may be further included so that the user can identify candidate objects corresponding to each identification information.
  • the user can select candidate objects to be masked in the target video through the masking selection interface, and can select candidate objects to be masked by checking the checkbox displayed next to the identification information list. That is, it is possible to selectively set candidate objects for masking, rather than collectively masking all objects included in the target video.
  • the above-described present invention can be embodied as computer readable codes on a medium on which a program is recorded.
  • the computer-readable medium may be one that continuously stores an executable program on a computer or temporarily stores it for execution or download.
  • the medium may be various recording means or storage means in the form of a combination of single or several hardware, and is not limited to a medium directly connected to a computer system, but may be distributed on a network. Examples of the medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks, And program instructions including ROM, RAM, flash memory, and the like.
  • examples of other media include an application store for distributing applications, a site for distributing or distributing various software, and a recording medium or storage medium managed by a server. Accordingly, the above detailed description should not be construed as limiting in all respects, but should be considered illustrative. The scope of the invention should be determined by rational interpretation of the appended claims, and all changes within the equivalent scope of the invention are included in the scope of the invention.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)

Abstract

La présente invention concerne un dispositif de masquage d'image et un procédé de masquage d'image. Selon un mode de réalisation de la présente invention, un procédé de masquage d'image peut comprendre les étapes suivantes consistant : à décoder une vidéo cible reçue afin de générer des images de trame sur une base par trame ; à extraire un vecteur de caractéristiques de trame correspondant à chaque image de trame à partir des images de trame, et à détecter des objets candidats à partir du vecteur de caractéristiques de trame ; à extraire un vecteur de caractéristiques d'objet correspondant à chaque objet candidat à partir du vecteur de caractéristiques de trame en utilisant des informations de localisation des objets candidats détectés ; à définir des informations d'identification correspondant aux objets candidats au moyen du vecteur de caractéristiques d'objet ; à suivre des objets candidats ayant les mêmes informations d'identification dans des images de trame continue, et à générer, pour chaque information d'identification, des informations de suivi correspondant aux objets candidats ; et lors de la réception d'une entrée de masquage, à extraire, au moyen des informations de suivi, des objets candidats identiques aux informations d'identification comprises dans l'entrée de masquage, et à effectuer le masquage sur les objets candidats extraits.
PCT/KR2019/013173 2018-12-20 2019-10-08 Dispositif de masquage d'image et procédé de masquage d'image Ceased WO2020130309A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020180165833A KR101972918B1 (ko) 2018-12-20 2018-12-20 영상 마스킹 장치 및 영상 마스킹 방법
KR10-2018-0165833 2018-12-20

Publications (1)

Publication Number Publication Date
WO2020130309A1 true WO2020130309A1 (fr) 2020-06-25

Family

ID=67807523

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2019/013173 Ceased WO2020130309A1 (fr) 2018-12-20 2019-10-08 Dispositif de masquage d'image et procédé de masquage d'image

Country Status (2)

Country Link
KR (1) KR101972918B1 (fr)
WO (1) WO2020130309A1 (fr)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111985419A (zh) * 2020-08-25 2020-11-24 腾讯科技（深圳）有限公司 视频处理方法及相关设备
CN112153422A (zh) * 2020-09-25 2020-12-29 连尚(北京)网络科技有限公司 视频融合方法和设备
CN112381092A (zh) * 2020-11-20 2021-02-19 深圳力维智联技术有限公司 跟踪方法、装置及计算机可读存储介质
US20220189036A1 (en) * 2020-12-11 2022-06-16 Electronics And Telecommunications Research Institute Contour-based privacy masking apparatus, contour-based privacy unmasking apparatus, and method for sharing privacy masking region descriptor
CN116664849A (zh) * 2023-05-18 2023-08-29 中关村科学城城市大脑股份有限公司 数据处理方法、装置、电子设备和计算机可读介质
WO2024212453A1 (fr) * 2023-04-13 2024-10-17 惠州市德赛西威汽车电子股份有限公司 Procédé et appareil de traitement vidéo, dispositif et support de stockage

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102147187B1 (ko) * 2019-12-03 2020-08-24 네오컨버전스 주식회사 딥러닝 기반 동영상에서 연속적으로 개인정보를 비식별화하기 위한 방법 및 장치
KR102202577B1 (ko) * 2019-12-09 2021-01-14 네오컨버전스 주식회사 딥러닝 기반 개인정보 비식별화 방법 및 장치
KR102123248B1 (ko) * 2020-01-30 2020-06-16 김선진 프라이버시 보호를 위한 얼굴인식 기반 실시간 영상처리 시스템
KR102415223B1 (ko) * 2020-06-03 2022-06-29 서울대학교산학협력단 동영상으로부터 보행자의 영역을 검출하기 위한 방법 및 이를 수행하기 위한 장치
CN111931654B (zh) * 2020-08-11 2024-11-22 精英数智科技股份有限公司 一种人员跟踪智能监测方法、系统和装置
KR102570184B1 (ko) * 2020-09-28 2023-08-25 경북대학교 산학협력단 보행 영상 기반의 보행 측정 방법 및 이의 장치
WO2022098074A1 (fr) * 2020-11-06 2022-05-12 카페24 주식회사 Procédé, dispositif et système pour protéger des droits de portrait
KR102754010B1 (ko) * 2022-10-31 2025-01-16 주식회사 인피닉스 딥러닝 기반의 개인정보 추적을 위한 비식별 방법 및 이를 수행하는 시스템
KR102621196B1 (ko) * 2023-07-10 2024-01-09 주식회사 웹게이트 개인 영상정보 보호 cctv 시스템

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004179709A (ja) * 2002-11-25 2004-06-24 Fuji Photo Film Co Ltd ディジタル・カメラおよびその制御方法
KR20080045347A (ko) * 2006-11-20 2008-05-23 포스데이타 주식회사 워터마크를 이용한 영상 생성, 모니터링 인증 및 관리 방법및 장치
JP2013210845A (ja) * 2012-03-30 2013-10-10 Secom Co Ltd 移動物体照合装置
KR101523740B1 (ko) * 2014-01-03 2015-05-28 (주)유디피 공간 매핑을 이용한 객체 추적 장치 및 방법
KR101517359B1 (ko) * 2013-12-27 2015-06-05 전남대학교산학협력단 객체 추적 방법 및 이를 이용한 객체 추적 장치

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004179709A (ja) * 2002-11-25 2004-06-24 Fuji Photo Film Co Ltd ディジタル・カメラおよびその制御方法
KR20080045347A (ko) * 2006-11-20 2008-05-23 포스데이타 주식회사 워터마크를 이용한 영상 생성, 모니터링 인증 및 관리 방법및 장치
JP2013210845A (ja) * 2012-03-30 2013-10-10 Secom Co Ltd 移動物体照合装置
KR101517359B1 (ko) * 2013-12-27 2015-06-05 전남대학교산학협력단 객체 추적 방법 및 이를 이용한 객체 추적 장치
KR101523740B1 (ko) * 2014-01-03 2015-05-28 (주)유디피 공간 매핑을 이용한 객체 추적 장치 및 방법

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111985419A (zh) * 2020-08-25 2020-11-24 腾讯科技（深圳）有限公司 视频处理方法及相关设备
CN112153422A (zh) * 2020-09-25 2020-12-29 连尚(北京)网络科技有限公司 视频融合方法和设备
CN112381092A (zh) * 2020-11-20 2021-02-19 深圳力维智联技术有限公司 跟踪方法、装置及计算机可读存储介质
US20220189036A1 (en) * 2020-12-11 2022-06-16 Electronics And Telecommunications Research Institute Contour-based privacy masking apparatus, contour-based privacy unmasking apparatus, and method for sharing privacy masking region descriptor
US12141988B2 (en) * 2020-12-11 2024-11-12 Electronics And Telecommunications Research Institute Contour-based privacy masking apparatus, contour-based privacy unmasking apparatus, and method for sharing privacy masking region descriptor
WO2024212453A1 (fr) * 2023-04-13 2024-10-17 惠州市德赛西威汽车电子股份有限公司 Procédé et appareil de traitement vidéo, dispositif et support de stockage
CN116664849A (zh) * 2023-05-18 2023-08-29 中关村科学城城市大脑股份有限公司 数据处理方法、装置、电子设备和计算机可读介质
CN116664849B (zh) * 2023-05-18 2024-01-16 中关村科学城城市大脑股份有限公司 数据处理方法、装置、电子设备和计算机可读介质

Also Published As

Publication number Publication date
KR101972918B1 (ko) 2019-08-20

Similar Documents

Publication Publication Date Title
WO2020130309A1 (fr) Dispositif de masquage d'image et procédé de masquage d'image
KR102167730B1 (ko) 영상 마스킹 장치 및 영상 마스킹 방법
JP4036051B2 (ja) 顔照合装置および顔照合方法
WO2022124701A1 (fr) Procédé de production d'une image étiquetée à partir d'une image originale tout en empêchant une fuite d'informations privées d'une image originale et serveur l'utilisant
KR101781358B1 (ko) 디지털 영상 내의 얼굴 인식을 통한 개인 식별 시스템 및 방법
WO2017204596A1 (fr) Procédé et dispositif de correction de contours faciaux
WO2020040391A1 (fr) Système combiné basé sur un réseau de couches profondes pour une reconnaissance de piétons et une extraction d'attributs
WO2015034269A1 (fr) Procédé et dispositif de traitement d'images
WO2021100919A1 (fr) Procédé, programme et système pour déterminer si un comportement anormal se produit, sur la base d'une séquence de comportement
WO2017090892A1 (fr) Caméra de génération d'informations d'affichage à l'écran, terminal de synthèse d'informations d'affichage à l'écran (20) et système de partage d'informations d'affichage à l'écran le comprenant
WO2018135906A1 (fr) Caméra et procédé de traitement d'image d'une caméra
WO2021162305A1 (fr) Procédé de fonctionnement d'un serveur de manière à fournir un service de plate-forme fondée sur une vidéo de sport
EP3459009A2 (fr) Procédé de quantification adaptative pour codage d'image d'iris
WO2019017720A1 (fr) Système de caméra permettant la protection de la confidentialité et procédé correspondant
WO2023277472A1 (fr) Procédé et dispositif électronique permettant de capturer une image d'un objet pour identifier un animal de compagnie
WO2023172031A1 (fr) Génération d'images de surveillance panoramique
WO2023158205A1 (fr) Élimination de bruit d'une image de caméra de surveillance au moyen d'une reconnaissance d'objets basée sur l'ia
WO2023080667A1 (fr) Traitement d'image wdr de caméra de surveillance par reconnaissance d'objets basée sur l'ia
KR20210036609A (ko) 영상 처리 장치 및 방법
WO2023204449A1 (fr) Procédé d'apprentissage et dispositif d'apprentissage pour la formation d'un réseau de masquage apte à masquer des données d'origine pour la confidentialité pour obtenir un masquage de restriction d'informations et procédé de test et dispositif de test utilisant ces derniers
WO2018008934A2 (fr) Procédé de quantification adaptative pour codage d'image d'iris
JPWO2008035411A1 (ja) 移動体情報検出装置、移動体情報検出方法および移動体情報検出プログラム
WO2025135266A1 (fr) Procédé et système de mesure de poussière fine sur la base d'une image
WO2023224377A1 (fr) Procédé de gestion d'informations d'un objet et appareil appliquant ledit procédé
WO2019124634A1 (fr) Procédé orienté syntaxe de suivi d'objet dans une vidéo comprimée

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19900236

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19900236

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 06/10/2021)

122 Ep: pct application non-entry in european phase

Ref document number: 19900236

Country of ref document: EP

Kind code of ref document: A1