WO2025186903A1 - Information processing device, information processing method, and recording medium - Google Patents

Abstract

This information processing device includes: an acquisition means for acquiring an image from a video; a detection means for detecting the position of a tracking target included in the image; a conversion means for converting position information related to the position of the tracking target into a feature amount indicating a feature of the position information; an update means for updating the feature amount by using a cross attention mechanism capable of cross-checking the tracking target between the plurality of images; and a restoration means for restoring the updated feature amount to position information. According to such an information processing device, it is possible to accurately track a tracking target included in a video.

Description

Information processing device, information processing method, and recording medium

This disclosure relates to the technical fields of information processing devices, information processing methods, and recording media.

A known device of this type is one that performs tracking of objects contained in video. For example, Patent Document 1 discloses generating a first feature vector indicating the position information of an object in a first image and a second feature vector indicating the position information of the object in a second image, and using the first feature vector and the second feature vector to generate correspondence information indicating the correspondence between the objects, thereby tracking the objects.

International Publication No. 2021/130951

The objective of this disclosure is to provide an information processing device, an information processing method, and a recording medium that aim to improve upon the technology disclosed in prior art documents.

One aspect of the information processing device disclosed herein comprises an acquisition means for acquiring images from a video, a detection means for detecting the position of a tracked object contained in the images, a conversion means for converting position information relating to the position of the tracked object into a feature quantity indicating the characteristics of the position information, an update means for updating the feature quantity using a cross-attention mechanism capable of matching the tracked object between multiple images, and a restoration means for restoring the updated feature quantity to the position information.

One aspect of the information processing method disclosed herein involves using at least one computer to acquire images from a video, detect the position of a tracked object contained in the images, convert the position information regarding the position of the tracked object into features indicating the characteristics of the position information, update the features using a cross-attention mechanism capable of matching the tracked object across multiple images, and restore the updated features to the position information.

One aspect of the recording medium of this disclosure is a computer program recorded on at least one computer that causes the computer to execute an information processing method that acquires images from a video, detects the position of a tracked target included in the images, converts position information regarding the position of the tracked target into feature quantities that indicate the characteristics of the position information, updates the feature quantities using a cross-attention mechanism that can match the tracked target between multiple images, and restores the updated feature quantities to the position information.

FIG. 2 is a block diagram showing a hardware configuration of a first information processing apparatus. FIG. 10 is a conceptual diagram illustrating an example of a tracking technique using query propagation. FIG. 2 is a block diagram showing a functional configuration of a first information processing apparatus. 10 is a flowchart showing the flow of a tracking process by the first information processing device. FIG. 2 is a block diagram showing the configuration of an intersection attention mechanism in the first information processing device. FIG. 10 is a plan view illustrating an example of an affinity matrix calculated by a cross-attention mechanism. FIG. 2 is a block diagram showing a functional configuration of a second information processing apparatus. FIG. 10 is a conceptual diagram illustrating a method for generating training data according to a comparative example. FIG. 10 is a conceptual diagram illustrating a method for generating learning data according to the second information processing device. FIG. 10 is a conceptual diagram illustrating query propagation in a learning operation by a second information processing device. 10 is a flowchart showing the flow of a learning operation by the second information processing device.

Below, embodiments of an information processing device, an information processing method, and a recording medium will be described with reference to the drawings.

First Embodiment
The first information processing apparatus will be described with reference to FIGS.

(Hardware configuration)
First, the hardware configuration of the first information processing apparatus will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the hardware configuration of the first information processing apparatus.

As shown in FIG. 1, the first information processing device 1 includes a processor 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, a storage device 14, an input device 15, and an output device 16. The processor 11, RAM 12, ROM 13, storage device 14, input device 15, and output device 16 are each connected via a data bus 17. Note that the data bus 17 may be an interface other than a data bus (for example, a LAN, USB, etc.).

The processor 11 loads a computer program. For example, the processor 11 is configured to load a computer program stored in at least one of the RAM 12, ROM 13, and storage device 14. Alternatively, the processor 11 may load a computer program stored in a computer-readable storage medium using a storage medium reading device (not shown). The processor 11 may also obtain (i.e., load) the computer program from a device (not shown) located outside the first information processing device 1 via a network interface. The processor 11 performs various processes by executing the loaded computer program. In particular, in this embodiment, when the processor 11 executes the loaded computer program, functional blocks related to the tracking process performed by the first information processing device 1 are realized within the processor 11. In other words, the processor 11 may function as a controller that executes each control in the first information processing device 1.

Processor 11 may be configured as, for example, a CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), or quantum processor. Processor 11 may be configured as one of these, or as multiple processors operating in parallel.

RAM 12 temporarily stores computer programs executed by processor 11. RAM 12 temporarily stores data that processor 11 uses temporarily while it is executing a computer program. RAM 12 may be, for example, D-RAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory). Also, other types of volatile memory may be used instead of RAM 12.

ROM 13 stores computer programs executed by processor 11. ROM 13 may also store fixed data. ROM 13 may be, for example, a P-ROM (Programmable Read Only Memory) or an EPROM (Erasable Read Only Memory). Also, other types of non-volatile memory may be used instead of ROM 13.

The storage device 14 stores data that the first information processing device 1 saves over the long term. The storage device 14 may operate as a temporary storage device for the processor 11. The storage device 14 may also store computer programs executed by the processor 11. The storage device 14 may include, for example, at least one of a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device.

The input device 15 is a device that receives input instructions from a user of the information processing device 1. The input device 15 may include, for example, at least one of a keyboard, a mouse, and a touch panel. The input device 15 may also be a device that allows voice input, for example, including a microphone. The input device 15 may also be configured as various types of terminals, such as a smartphone, tablet, or laptop computer.

The output device 16 is a device that outputs information related to the information processing device 1 to the outside. For example, the output device 16 may be a display device (e.g., a display or digital signage) that can display information related to the information processing device 1. The output device 16 may also be a speaker or the like that can output information related to the first information processing device 1 as audio. The output device 16 may be configured as various terminals, such as a smartphone, tablet, or laptop computer.

The first information processing device 1 may be configured to include some of the components described in FIG. 1. For example, the first information processing device 1 may be configured to include a processor 11, RAM 12, and ROM 13. In this case, the storage device 14, input device 15, and output device 16 may each be configured as an external device connected to the first information processing device 1. Furthermore, some of the calculation functions of the first information processing device 1 may be realized by an external server, cloud, etc.

(Tracking method)
Next, a tracing process technique executed by the first information processing device 1 will be described with reference to Fig. 2. Fig. 2 is a conceptual diagram showing an example of a tracing technique using query propagation.

In FIG. 2, the first information processing device 1 is configured to be able to execute a tracking process for tracking a tracking target included in a video. The tracking target may be, for example, a person or an animal, or an object such as luggage or a car. In the tracking process executed by the first information processing device, a feature based on the position information of the tracking target detected from the video is acquired as a "detection query." This detection query is then used to update a "tracking query," which is a query for tracking the target. The first information processing device 1 tracks the tracking target by propagating this tracking query in chronological order. Note that these queries are set for each tracking target. Therefore, if a video includes multiple tracking targets, the queries corresponding to each of the multiple tracking targets are updated.

For example, in the example shown in Figure 2, objects A and B are detected from a frame captured at time T1. Therefore, the detection query at time T1 includes detection queries for objects A and B. The tracking query is then updated using the detection query at time T1, and as a result, the tracking query at time T2 includes tracking queries for objects A and B.

Subsequently, a new object C is detected in the frame captured at time T2. Therefore, the detection query at time T2 includes the detection query for object C. The tracking query is then updated using the detection query at time T2. As a result, the tracking query at time T3 includes the tracking queries for objects A and B that were included in the tracking query at time T2 (in other words, propagated from the previous time), as well as the tracking query for the newly detected object C.

Note that in the frame captured at time T3, objects A, B, and C are each detected, but in the frame captured at time T4, only objects A and C are detected, and object B is not. As a result, the tracking query for object B has disappeared from the tracking query at time T4, and tracking queries for objects A and C are included.

(Functional configuration)
Next, the functional configuration for executing the above-mentioned tracking process will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the functional configuration of the first information processing apparatus.

As shown in FIG. 3, the first information processing device 10 includes, as components for realizing its functions, an image acquisition unit 110, a target position detection unit 120, a feature conversion unit 130, a feature update unit 140, a location information restoration unit 150, and a storage unit 155. Each of the image acquisition unit 110, target position detection unit 120, feature conversion unit 130, feature update unit 140, and location information restoration unit 150 may be a processing block realized by the processor 11 described above (see FIG. 1). The storage unit 155 may be realized by the storage device 14 described above (see FIG. 1), etc.

The image acquisition unit 110 is configured to be able to acquire videos and images. More specifically, the image acquisition unit 110 may acquire images of each frame constituting a video sequentially in chronological order. Alternatively, it may acquire images arranged in chronological order. For example, images may be acquired from each frame constituting a video at predetermined intervals. The image acquisition unit 110 may also be configured to acquire images in real time while capturing a video with a camera. The images acquired by the image acquisition unit 110 are configured to be output to the target position detection unit 120.

The target position detection unit 120 is configured to be able to detect the position of the tracking target from the image acquired by the image acquisition unit 110. The target position detection unit 120 may be configured to, for example, detect a person included in the image and output position information indicating the position of the detected person. Note that if the image includes multiple tracking targets, the target position detection unit 120 may detect position information for each of the multiple tracking targets. The target position detection unit 120 may be configured as a trained detection model made up of, for example, a neural network. The position information of the tracking target detected by the target position detection unit 120 is configured to be output to the feature conversion unit 130.

The feature conversion unit 130 is configured to be able to convert the position information of the tracked target detected by the target position detection unit 120 into feature amounts. The feature amounts here may be feature vectors indicating the position information of the tracked target. The feature conversion unit 130 may be configured as an encoder equipped with multiple feature extraction blocks. The feature conversion unit 130 may be constructed, for example, with approximately three fully connected layers in a neural network. The feature amounts converted by the feature conversion unit 130 may be used, for example, as detection queries (see Figure 2) used in tracking processing. The feature amounts converted by the feature conversion unit 130 are configured to be output to the feature update unit 140.

The feature update unit 140 is configured to be able to update the features converted by the feature conversion unit 130 using a cross-attention mechanism 200. The cross-attention mechanism 200 has the function of comparing the position information of the tracked target between multiple images. Specifically, the cross-attention mechanism 200 has the function of associating targets included in each of consecutive images acquired in time series to determine whether they are the same person. The specific configuration of the cross-attention mechanism 200 will be explained in detail later. The features updated by the feature update unit 140 (hereinafter referred to as "updated features") may be used, for example, as a tracking query (see Figure 2) used in the tracking process. In this case, the updated features may be temporarily stored in the memory unit 155, which stores tracking queries. The updated features are also configured to be output to the position information restoration unit 150.

The location information restoration unit 150 is configured to be able to restore the updated features updated by the feature update unit 140 to location information. The location information restoration unit 150 may be configured as a decoder equipped with multiple feature extraction blocks. The location information restoration unit 150 may be constructed, for example, with approximately three fully connected layers in a neural network. The location information restoration unit 150 may also have a function to output the restored location information.

The storage unit 155 is configured to store feature amounts and location information for each tracking target. For example, the storage unit 155 may be configured to store detection queries and tracking queries for each tracking target. For example, the storage unit 155 may have multiple memory areas, and store tracking queries for tracking a single person in one memory area so that they are accumulated in chronological order. The storage unit 155 may also be configured to store feature amounts and location information for each tracking target, linked to the corresponding ID.

(Tracking process flow)
Next, the flow of the tracking process executed by the first information processing device 1 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the flow of the tracking process by the first information processing device.

As shown in FIG. 5, when the tracking process by the first information processing device 1 begins, the image acquisition unit 110 first acquires an image from the video (step S101). Then, the target position detection unit 120 detects the position of the tracking target contained in the image from the image acquired by the image acquisition unit 110 (step S102).

Next, the feature conversion unit 130 converts the position information of the tracked target detected by the target position detection unit 120 into a feature (step S103). Then, the feature update unit 140 updates the feature converted by the feature conversion unit 130 using the intersection attention mechanism 200 (step S104).

Then, the location information restoration unit 150 restores the updated feature amounts updated by the feature amount update unit 140 into location information (step S105). Then, the first information processing device 1 determines whether to end the tracking process (step S106).

If the tracking process is not to be ended (step S106: NO), the process may be executed again from step S101. That is, the next frame image may be acquired, and the above-described process may be executed repeatedly. On the other hand, if the tracking process is not to be ended (step S106: YES), the series of processes will end.

(Cross-Attention Mechanism)
Next, the configuration and operation of the intersection attention mechanism 200 will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the configuration of the intersection attention mechanism in the first information processing apparatus.

As shown in FIG. 5, the cross-attention mechanism 200 includes three feature embedding units 210, 220, and 230 corresponding to the query, key, and value, respectively, a matrix multiplication unit 240, a normalization unit 250, a matrix multiplication unit 260, a residual processing unit 270, and a memory update unit 280.

The feature embedding processor 210 is configured to extract a query from the feature values at time t (i.e., the feature values corresponding to the frame captured at time t) input from the feature converter 130. The feature embedding processor 220 is configured to extract a key from the feature values at time t-τ calculated in a past tracking process (i.e., the feature values corresponding to the frame captured at time t-τ, prior to time t). The feature embedding processor 230 is configured to extract a value from the feature values at time t-τ calculated in a past tracking process. The query and key are configured to be output to the matrix multiplication processor 240. On the other hand, the value is configured to be output to the matrix multiplication processor 260.

The matrix multiplication calculation unit 240 is configured to calculate a weight (Attention Weight) indicating the correlation between the query and the key by calculating the matrix product of the query and the key. In other words, the matrix calculation unit 240 is configured to calculate a weight indicating the correlation between the feature corresponding to the frame captured at time t and the feature corresponding to the frame captured at time t-τ. For example, the matrix multiplication calculation unit 240 may calculate (use) an affinity matrix (Affinity Matrix) in which the vertical axis represents the feature corresponding to the frame captured at time t and the horizontal axis represents the feature corresponding to the frame captured at time t-τ as the weight (Attention Weight) of the cross-attention mechanism 200.

The normalization unit 250 is configured to be able to perform normalization processing on the weights calculated by the matrix multiplication unit 240. The normalization unit 250 may, for example, perform processing to normalize the similarity matrix calculated by the matrix multiplication unit 240 using a cross-softmax function. The weights normalized by the normalization unit 250 are configured to be output to the matrix multiplication unit 260.

The matrix multiplication unit 260 is configured to perform processing to reflect the weight in the value by calculating the matrix product of the output from the normalization unit 250 and the value. Note that the matrix product in this embodiment may typically be a tensor product (in other words, a direct product). For example, the matrix product may be a Kronecker product. The calculation result of the matrix multiplication unit 260 is configured to be output to the residual processing unit 270.

The residual processing unit 270 is configured to perform residual processing on the calculation results of the matrix multiplication calculation unit 260. This residual processing may be a process of adding the calculation results of the matrix multiplication calculation unit 260 and the feature values input to the cross-attention mechanism 200 (specifically, the feature values at time t). This is to prevent the feature values from not being generated as the calculation results of the cross-attention mechanism 200 even if a correlation is not calculated. For example, if 0 is calculated as the correlation (weight), the value will be multiplied by that 0, causing the feature value in the calculation results of the matrix calculation unit 260 to become 0 (disappear). To prevent this, the residual processing unit 270 performs the residual processing described above. The calculation results of the residual processing unit 270 are output from the cross-attention mechanism 200 as the updated feature values at time t.

The memory update unit 280 updates the stored feature quantities corresponding to the tracked object. The memory update unit 280 may update only the feature quantities stored in the memory means corresponding to the updated feature quantities output by the matrix multiplication calculation unit 260, or may overwrite the feature quantities output by the calculation results of the residual processing unit 270 in the memory means to update them. For example, the tracked object may be identified by weights calculated from the query and key by the matrix multiplication calculation unit 240, and it may be determined which tracked object to update from the multiple tracked objects stored in the memory unit 155. Furthermore, the updated feature quantities calculated by the matrix multiplication calculation unit 260 from the normalized weights and values may be determined as the updated quantities for the feature quantities of the tracked object stored in the memory unit 155.

Note that the first information processing device 1 essentially focuses on the similarity between the tracking process and the operations performed by the cross-attention mechanism 200, and can be said to perform an operation to update features using information generated when matching objects. For example, the tracking process involves a process to detect the tracked object, a process to match the tracked object, and a process to update the tracked object detection results. On the other hand, the cross-attention mechanism 200 involves a process to extract features related to the tracked object, a process to calculate weights, and a process to update the features related to the tracked object. The first information processing device 1 essentially reuses the process of calculating weights in the cross-attention mechanism 200 as the process of matching the tracked object in the tracking process. In other words, the first information processing device 1 essentially reuses the process of matching the tracked object in the tracking process as the process of calculating weights in the cross-attention mechanism 200. Therefore, it can also be said that the first information processing device 1 realizes the operations of detecting an object, matching the object, and updating the detection results using the cross-attention mechanism 200.

More specifically, as described above, the intersection attention mechanism 200 uses the feature quantity corresponding to the frame captured at time t as a query, and obtains and uses the feature quantities contained in frames captured up to time t-τ before time t as keys and values from the memory unit 160, which stores these feature quantities, to track the target. The memory update unit 280 may also update the memory means by overwriting the feature quantity output by the calculation results of the residual processing unit 270. Alternatively, the target may be identified from the weights calculated by the matrix multiplication calculation unit 240, and the feature quantity corresponding to the ID of the identified target, which is stored in the memory means, may be updated using the updated feature quantity output by the matrix multiplication calculation unit 260. In this way, it is possible to accurately track a target included in a video using a relatively simple algorithm.

(similarity matrix)
Next, the similarity matrix calculated by the above-described intersection attention mechanism 200 will be specifically described with reference to Fig. 6. Fig. 6 is a plan view showing an example of the similarity matrix calculated by the intersection attention mechanism.

As shown in FIG. 6, the similarity matrix AM used as a weight by the cross attention mechanism 200 is information indicating the correspondence between the tracking target O _t- τ at time t-τ and the tracking target O _t at time t. For example, the similarity matrix AM is information indicating that (1) a first tracking target O t _{-τ among} the multiple tracking targets O t- _τ corresponds to a first tracking target O _t among the multiple tracking targets O t (that is, both are the same person), (2) a second tracking target O _{t-τ among} the multiple tracking targets O t- _τ corresponds to a second tracking target _{O t among} the multiple tracking targets O t, ..., (N) an Nth tracking target O _t-τ among the multiple tracking targets O t- _τ corresponds to an Nth tracking target O _t among the multiple tracking targets O t. Note that the similarity matrix AM is information indicating the correspondence between the tracking target O _t-τ and the tracking target O _t , and therefore may be referred to as correspondence information.

Specifically, the similarity matrix AM can be considered to be a matrix whose vertical axis corresponds to the vector components of the feature vector CV _t-τ and whose horizontal axis corresponds to the vector components of the feature vector CV _t . Therefore, the size of the vertical axis of the similarity matrix AM is the size of the feature vector CV _t-τ , which corresponds to the size of the image captured at time t-τ (i.e., the number of pixels). Similarly, the size of the horizontal axis of the similarity matrix AM is the size of the feature vector CV _t , which corresponds to the size of the image captured at time t (i.e., the number of pixels). In other words, the similarity matrix AM can be considered to be a matrix whose vertical axis corresponds to the detection result of the tracking target O _t-τ reflected in the image at time t-τ (i.e., the detected position of the tracking target O _t-τ ) and whose horizontal axis corresponds to the detection result of the tracking target O _t reflected in the image at time t (i.e., the detected position of the tracking target O _t ). In this case, an element of the similarity matrix AM reacts (typically has a non-zero value) at a position where a vector component on the vertical axis corresponding to a certain tracking target O _{t-τ intersects} with a vector component on the horizontal axis corresponding to the same tracking target O t. In other words, an element of the similarity matrix AM reacts at a position where a detection result for tracking target O _t-τ on the vertical axis intersects with a detection result for tracking target O _t on the horizontal axis. In other words, the similarity matrix AM is typically a matrix in which the value of an element at a position where a vector component corresponding to tracking target O _{t- τ} included in feature vector CV t- _τ intersects with a vector component corresponding to the same tracking target O _t included in feature vector CV t is a value obtained by multiplying both vector components (i.e., a non-zero value), while the values of the other elements are 0.

For example, in the example shown in Fig. 5, the elements of the similarity matrix _{AM react} at positions where vector components corresponding to tracking target O#k (where k is the number of detected tracking targets O, and in the example shown in Fig. 5, k = 1, 2, 3, or 4) included in the feature vector CV t- _τ intersect with vector components corresponding to the same tracking target O#k included in the feature vector CV t. In other words, the elements of the similarity matrix AM react at positions where the detection result of tracking target O#k reflected in the image captured at time t-τ intersects with the detection result of tracking target O#k reflected in the image captured at time t.

Conversely, if the elements of the similarity matrix AM do not react (typically become 0) at the position where the vector component corresponding to the tracking target O _{t-τ included} in the feature vector CV _{t- τ} intersects with the vector component corresponding to the same tracking target O t included in the feature vector CV t, it is estimated that the tracking target O _t-τ that was reflected in the image captured at time t-τ is not reflected in the image captured at time t (for example, it has moved outside the angle of view of the camera).

In this way, the similarity matrix AM can be used as information indicating the correspondence between the tracking target O _t-τ and the tracking target O _t . In other words, the similarity matrix AM can be used as information indicating the result of matching the tracking target O _t-τ reflected in the image captured at time t-τ with the object O _t reflected in the image captured at time t. Therefore, the similarity matrix AM can be used as information for tracking the position of the tracking target O _t-τ reflected in the image captured at time t-τ within the image captured at time t. By using the similarity matrix AM in this way, it is possible to accurately perform tracking processing of the tracking target included in the video.

(Technical effect)
Next, the technical effects obtained by the first information processing device 1 will be described.

1 to 6, in the first information processing device 1, feature quantities related to the position information of an object in an image acquired from a video are updated using a cross-attention mechanism 200 having the function of matching objects. In the first information processing device 1, the operation of this cross-attention mechanism 200 is used to perform tracking processing for the tracked object. In this way, tracking processing can be performed more appropriately compared to when the cross-attention mechanism 200 described in this embodiment is not used. For example, when attempting to use a self-attention mechanism for tracking processing, learning is required so that the weights in the self-attention mechanism react strongly between the same tracked objects. However, realizing such a configuration requires a large number of self-attention mechanisms, which poses a technical problem in that the algorithm used for tracking processing becomes complicated. However, by using the cross-attention mechanism 200 described in this embodiment, the tracking processing algorithm can be constructed with a simple structure, making it possible to achieve highly accurate tracking processing while suppressing computational costs.

Second Embodiment
The second information processing device 1 will be described with reference to Figures 7 to 11. The second information processing device 1 differs in some configurations and operations from the first information processing device 1 described above, but other parts may be similar to the first information processing device 1. Therefore, the following will describe in detail the parts that differ from the first embodiment, and will omit explanations of other overlapping parts as appropriate.

(Functional configuration)
First, the functional configuration of the second information processing device 1 will be described with reference to Fig. 7. Fig. 7 is a block diagram showing the functional configuration of the second information processing device.

As shown in FIG. 7, the second information processing device 1 includes, as components for realizing its functions, an image acquisition unit 110, a target position detection unit 120, a feature conversion unit 130, a feature update unit 140, a position information restoration unit 150, and a learning unit 160. That is, the second information processing device 1 further includes a learning unit 160 in addition to the configuration already described in the first embodiment (see FIG. 3). Note that the learning unit 160 may be a processing block realized by the above-mentioned processor 11 (see FIG. 1).

The learning unit 160 is configured to be able to perform learning related to the tracking process of the tracked object executed by the second information processing device 1. More specifically, the learning unit 160 may perform machine learning on the tracking model 50 that performs the tracking process (i.e., a model having the functions of the object position detection unit 120, feature conversion unit 130, feature update unit 140, and object information restoration unit 140) to enable tracking with higher accuracy. The learning unit 160 may perform learning related to the operation of the cross-attention mechanism 200 used by the feature update unit 140. For example, the learning unit 160 may perform learning so that the operation of matching tracked objects performed by the cross-attention mechanism 200 can be performed more accurately. Specifically, the learning unit 160 may learn so that the similarity matrix AM used by the cross-attention mechanism 200 reacts more strongly to the same tracked object. The specific learning method used by the learning unit 160 will be explained in detail below.

(Learning method)
Next, the learning technique executed by the learning unit will be specifically described with reference to Fig. 8 to Fig. 10. Fig. 8 is a conceptual diagram showing a learning data generation technique according to a comparative example. Fig. 9 is a conceptual diagram showing a learning data generation technique according to a second information processing device. Fig. 10 is a conceptual diagram showing query propagation in the learning operation by the second information processing device.

In Figure 8, we will first explain a learning method according to a comparative example. In the learning method according to the comparative example, multiple videos are used as learning data. Specifically, multiple videos are each converted into mini-batches and used as learning data. When attempting to apply this type of learning method to learning tracking processing using query propagation, it is not possible to use many frames for learning due to memory limitations, for example, and as a result, it is not possible to obtain appropriate learning results.

On the other hand, as shown in FIG. 9, the learning unit 160 in the second information processing device 1 performs mini-batch conversion on a single video and uses the result as training data. For example, the learning unit 160 may use all of the video frames included in the mini-batch for training the query propagation of the tracking model 50. The training data may include each frame included in a single video and ground truth data indicating the correspondence between the tracked targets captured in each frame (i.e., which people are the same person).

As shown in Figure 10, the learning unit 160 uses multiple frames (frame t1, frame t2, frame t3, ...) arranged in chronological order from a single video as learning data. In this way, it is possible to significantly increase the number of frames that can be used to learn query propagation in the tracking model 50, and it is also possible to learn query propagation by the tracking model 50 in chronological order.

(Learning operation flow)
Next, the flow of the learning operation executed by the second information processing device 1 (i.e., the operation when the learning unit 160 learns the tracking model 50) will be described with reference to Fig. 11. Fig. 11 is a flowchart showing the flow of the learning operation by the second information processing device.

As shown in FIG. 11, when the learning operation by the second information processing device 10 begins, the learning unit 160 first batch-converts one video to create learning data (step S201). The learning unit 160 then inputs the learning data 202 into the tracking model 50 (step S202).

Next, the learning unit 160 compares the output result of the tracking model 50 with the ground truth data to calculate a loss function (step S203). Then, the learning unit 160 calculates the gradient of the loss function (step S204).

Next, the learning unit 160 updates the parameters of the tracking model based on the calculated gradient so as to reduce the loss function (step S205). After that, the learning unit 160 determines whether learning has been performed using all frames of the training data (step S206).

If not all frames have been used for learning (step S206: NO), the learning unit 160 starts processing again from step S202. That is, the learning unit 160 repeats the process from inputting images, which are learning data, into the tracking model 50 to updating the parameters. On the other hand, if all frames have been used for learning (step S206: YES), the learning unit 160 determines that learning has ended and saves the learned model (step S207).

(Technical effect)
Next, the technical effects obtained by the second information processing device 1 will be described.

As described in Figures 6 to 11, in the second information processing device 1, multiple frames included in one video are converted into one mini-batch for learning. In this way, it is possible to significantly increase the number of frames that can be used in learning query propagation for the tracking model 50, compared to, for example, when multiple videos are each converted into mini-batches. Note that there is no particular limit on the number of frames in the tracking process performed by the second information processing device 1. Therefore, by realizing learning using a large number of frames (in other words, long-term time-series learning), it is possible to effectively improve tracking accuracy.

The information processing device 1 according to each of the above-described embodiments can be applied to various systems that perform tracking processing on an object. For example, the information processing device 1 can be applied to a gateless authentication system that tracks an object passing through a predetermined area and performs authentication processing using biometric information (e.g., facial information, iris information, etc.) of the object being tracked.

The scope of each embodiment also includes a processing method in which a program that operates the configuration of each embodiment to realize the functions of the above-mentioned embodiments is recorded on a recording medium, the program recorded on the recording medium is read as code, and the program is executed on a computer. In other words, computer-readable recording media are also included in the scope of each embodiment. Furthermore, each embodiment includes not only the recording medium on which the above-mentioned program is recorded, but also the program itself.

The recording medium may be, for example, a floppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM, magnetic tape, non-volatile memory card, or ROM. Furthermore, the scope of each embodiment is not limited to programs recorded on the recording medium that execute processing by themselves, but also includes programs that execute processing by running on an OS in conjunction with other software or expansion board functions. Furthermore, the program itself may be stored on a server, and part or all of the program may be downloadable from the server to a user terminal. The program may also be provided to the user in, for example, a SaaS (Software as a Service) format.

<Additional Notes>
The above-described embodiment may be further described as follows, but is not limited to the following.

(Appendix 1)
The information processing device described in Supplementary Note 1 is an information processing device that includes an acquisition means for acquiring an image from a video, a detection means for detecting the position of a tracked object included in the image, a conversion means for converting position information regarding the position of the tracked object into a feature quantity indicating the characteristics of the position information, an update means for updating the feature quantity using a cross-attention mechanism that can match the tracked object between a plurality of the images, and a restoration means for restoring the updated feature quantity to the position information.

(Appendix 2)
The information processing device described in Appendix 2 is the information processing device described in Appendix 1, in which the cross-attention mechanism matches the tracked target using a weight calculated from a first feature that is the feature related to a first image and a second feature that is the feature related to a second image that was taken before the first image.

(Appendix 3)
The information processing device described in Supplementary Note 3 is the information processing device described in Supplementary Note 2, in which the cross-attention mechanism matches the tracked target by using a similarity matrix obtained by calculating a matrix product of the first feature and the second feature as the weight.

(Appendix 4)
The information processing device described in Supplementary Note 4 is the information processing device described in any one of Supplementary Notes 1 to 3, further comprising: a storage means capable of storing the feature; and a storage update means that updates the feature stored in the storage means based on the feature updated by the update means.

(Appendix 5)
The information processing device described in Supplementary Note 5 is the information processing device described in any one of Supplementary Notes 1 to 3, further including a learning unit that converts multiple frames included in one video into one mini-batch to use as training data, and performs learning related to matching of the tracked target.

(Appendix 6)
The information processing method described in Supplementary Note 6 is an information processing method that, by at least one computer, acquires an image from a video, detects a position of a tracked target included in the image, converts position information regarding the position of the tracked target into a feature amount indicating characteristics of the position information, updates the feature amount using a cross-attention mechanism that can match the tracked target between a plurality of the images, and restores the updated feature amount to the position information.

(Appendix 7)
The recording medium described in Supplementary Note 7 is a recording medium having recorded thereon a computer program for causing at least one computer to execute an information processing method of acquiring images from a video, detecting a position of a tracked target included in the images, converting position information regarding the position of the tracked target into feature quantities indicating characteristics of the position information, updating the feature quantities using a cross-attention mechanism capable of matching the tracked target between a plurality of the images, and restoring the updated feature quantities to the position information.

(Appendix 8)
The computer program described in Supplementary Note 8 is a computer program that causes at least one computer to execute an information processing method of acquiring images from a video, detecting a position of a tracked target included in the images, converting position information regarding the position of the tracked target into features indicating characteristics of the position information, updating the features using a cross-attention mechanism that can match the tracked target between a plurality of the images, and restoring the updated features to the position information.

(Appendix 9)
The tracking device described in Supplementary Note 9 is a tracking device including: an acquisition means for acquiring an image from a video; a detection means for detecting a position of a tracking target included in the image; a conversion means for converting position information regarding the position of the tracking target into a feature quantity indicating characteristics of the position information; an update means for updating the feature quantity using a cross-attention mechanism capable of matching the tracking target between a plurality of the images; a restoration means for restoring the updated feature quantity to the position information; and a tracking means for tracking the tracking target based on the restored position information.

(Appendix 10)
The tracking method described in Supplementary Note 10 is a tracking method that, by at least one computer, acquires an image from a video, detects a position of a tracked object included in the image, converts position information regarding the position of the tracked object into a feature amount indicating characteristics of the position information, updates the feature amount using a cross-attention mechanism that can match the tracked object between a plurality of the images, restores the updated feature amount to the position information, and tracks the tracked object based on the restored position information.

(Appendix 11)
The recording medium described in Supplementary Note 11 is a recording medium having recorded thereon a computer program for causing at least one computer to execute a tracking method of acquiring images from a video, detecting a position of a tracked target included in the images, converting position information regarding the position of the tracked target into feature amounts indicating characteristics of the position information, updating the feature amounts using a cross-attention mechanism capable of matching the tracked target between a plurality of the images, restoring the updated feature amounts to the position information, and tracking the tracked target based on the restored position information.

(Appendix 12)
The computer program described in Supplementary Note 12 is a computer program that causes at least one computer to execute a tracking method of acquiring images from a video, detecting a position of a tracked target included in the images, converting position information regarding the position of the tracked target into feature amounts indicating characteristics of the position information, updating the feature amounts using a cross-attention mechanism that can match the tracked target across a plurality of the images, restoring the updated feature amounts to the position information, and tracking the tracked target based on the restored position information.

This disclosure may be modified as appropriate within the scope of the claims and the spirit or concept of the invention as can be read from the entire specification, and information processing devices, information processing methods, and recording media incorporating such modifications are also included within the technical concept of this disclosure.

10 Information processing device 11 Processor 12 RAM
13 ROM
14 Storage device 15 Input device 16 Output device 17 Data bus 50 Tracking model 110 Image acquisition unit 120 Target position detection unit 130 Feature conversion unit 140 Feature update unit 150 Position information restoration unit 155 Storage unit 160 Learning unit 200 Intersection attention mechanism 210 Query 220 Key 230 Value 240 Matrix multiplication unit 250 Normalization unit 260 Matrix multiplication unit 270 Residual processing unit 280 Memory update unit

Claims (7)

an acquisition means for acquiring images from the video;
a detection means for detecting a position of a tracking target included in the image;
a conversion means for converting position information relating to the position of the tracked object into a feature quantity indicating a feature of the position information;
updating means for updating the feature amount using a cross-attention mechanism capable of matching the tracked object between a plurality of the images;
a restoration means for restoring the updated feature amount to the position information;
An information processing device comprising: the cross-attention mechanism matches the tracked target using a weight calculated from a first feature amount that is the feature amount related to a first image and a second feature amount that is the feature amount related to a second image captured before the first image;
The information processing device according to claim 1 . the cross-attention mechanism matches the tracked target by using a similarity matrix obtained by calculating a matrix product of the first feature amount and the second feature amount as the weight;
The information processing device according to claim 2 . a storage means capable of storing the feature amount;
a storage update means for updating the feature quantity stored in the storage means based on the feature quantity updated by the update means;
The information processing device according to claim 1 , further comprising: The tracking system further includes a learning unit that converts a plurality of frames included in one video into one mini-batch as training data and performs training related to matching of the tracked object.
The information processing device according to claim 1 . by at least one computer,
Get images from videos,
Detecting the position of a tracked object included in the image;
converting location information relating to the location of the tracked object into a feature quantity indicating a feature of the location information;
updating the feature amount using a cross-attention mechanism capable of matching the tracked object between the plurality of images;
restoring the updated feature amount to the position information;
Information processing methods. At least one computer
Get images from videos,
Detecting the position of a tracked object included in the image;
converting location information relating to the location of the tracked object into a feature quantity indicating a feature of the location information;
updating the feature amount using a cross-attention mechanism capable of matching the tracked object between the plurality of images;
restoring the updated feature amount to the position information;
A recording medium on which a computer program for executing an information processing method is recorded.