WO2025115813A1 - Behavior prediction device and behavior prediction method - Google Patents

Abstract

A behavior prediction device according to one aspect of the present invention comprises: a detection unit that analyzes an image obtained by imaging a region including a second area adjacent to a first area and detects a plurality of passersby appearing in the image; a prediction unit that obtains an entry index indicating the possibility that each of the passersby will enter the first area from the second area; and an aggregation unit that aggregates the entry indices of the passersby and determines a synthesis index indicating the possibility that at least one of the plurality of passersby will enter the first area from the second area.

Description

BEHAVIOR PREDICTION DEVICE AND BEHAVIOR PREDICTION METHOD

The present disclosure relates to a behavior prediction device and a behavior prediction method.
This application claims priority based on Japanese Application No. 2023-201098 filed on November 28, 2023, and incorporates by reference all of the contents of said Japanese application.

Patent Document 1 describes a system that detects pedestrians waiting to cross the crosswalk by analyzing the line of sight of people in an image captured by a camera that captures the waiting area of the crosswalk, and when a waiting pedestrian is detected, controls pedestrian and vehicle traffic lights to allow the pedestrian to cross the crosswalk.

JP 2017-208141 A

A behavior prediction device according to one aspect includes a detection unit that analyzes an image captured of an area including a second area surrounding a first area to detect multiple passersby appearing in the image, a prediction unit that calculates an entry index indicating the likelihood of each passerby entering the first area, and an aggregation unit that aggregates the entry indexes of each passerby to calculate a composite index indicating the likelihood that at least one of the multiple passersby will enter the first area.

FIG. 1 is a schematic configuration diagram of a behavior prediction system according to an embodiment. FIG. 2 is a diagram showing an imaging area. FIG. 3 is a block diagram showing a functional configuration of the behavior prediction device. Fig. 4A shows an example of one frame included in a moving image, and Fig. 4B shows an example of a frame including position information of a passerby. FIG. 5 shows an example of a movement trajectory of a passerby. FIG. 6 is a diagram for explaining an example of a method for determining the entry probability of each passerby. FIG. 7 is a diagram for explaining another example of a method for determining the entry probability of each passerby. FIG. 8 is a block diagram illustrating an example of the hardware configuration of the behavior prediction device. FIG. 9 is a flowchart illustrating a behavior prediction method according to an embodiment.

[Problem to be solved by this disclosure]
In order to perform traffic control efficiently, it is important to predict the behavior of pedestrians. For example, as described in the above-mentioned Patent Document 1, if it is possible to predict that pedestrians will move to an area including a pedestrian crossing, it is possible to control traffic lights to allow pedestrians to cross safely. However, in actual traffic environments, there are often many pedestrians. Even in such situations, it is desirable to be able to predict the entry of pedestrians into a specific area.

[Effects of the present disclosure]
According to the present disclosure, in a scene where a plurality of passersby are present, it is possible to predict the entry of passersby into a specific area.

[Description of the embodiments of the present disclosure]
First, the contents of the embodiments of the present disclosure will be listed and described.

[1] A behavior prediction device according to one aspect of the present disclosure includes a detection unit that analyzes an image captured of an area including a second area surrounding a first area to detect multiple passersby appearing in the image, a prediction unit that calculates an entry index indicating the likelihood of each passerby entering the first area, and an aggregation unit that aggregates the entry indexes of each passerby to calculate a composite index indicating the likelihood that at least one of the multiple passersby will enter the first area.

This behavior prediction device aggregates the indicators of each passerby entering the first area to determine the indicator of at least one of multiple passersby entering the first area, so that in a scene where multiple passersby are present, it is possible to predict whether at least one of the multiple passersby will enter the first area.

[2] In the behavior prediction device of [1] above, the first area may be an area including a crosswalk with traffic lights installed, and may further include a control unit that controls the traffic lights to allow multiple pedestrians to cross the crosswalk when the composite index is higher than a reference value. In this case, the traffic lights are controlled when there are pedestrians attempting to cross the crosswalk. Therefore, the pedestrians can cross safely.

[3] In the behavior prediction device described in [1] or [2] above, the detection unit may identify the movement trajectory of each of a plurality of passersby based on the position information of the passersby, and the prediction unit may obtain an entry indicator for each passerby based on the movement trajectory. By using the movement trajectory of the passersby, it is possible to predict with high accuracy whether or not the passersby will enter the first area.

[4] In the behavior prediction device described in any one of [1] to [3] above, the prediction unit may input the movement trajectory of each identified passerby and obtain the entry index of each passerby using a prediction model that has been machine-learned to output the entry index of each passerby. By using the prediction model, it is possible to predict with high accuracy whether or not a passerby will enter the first area.

[5] In the behavior prediction device described in any one of [1] to [3] above, the detection unit may identify the positions of multiple passersby, and the prediction unit may obtain an entry index for each passerby based on the distance between each passerby and the first area. The closer a passerby is to the first area, the more likely the passerby is to enter the first area. Therefore, by obtaining an entry index for each passerby based on the distance between each passerby and the first area, it is possible to predict whether or not the passerby will enter the first area.

[6] In the behavior prediction device described in [5] above, the prediction unit may further identify attributes of multiple passersby, and may obtain an entry index for each passerby based on the distance and the attributes of each passerby. Since the behavior of passersby may have characteristic patterns depending on the attributes, the prediction accuracy of the entry index for each passerby can be improved by obtaining the entry index using the attributes of each passerby.

[7] In the behavior prediction device described in any one of [1] to [3] above, the detection unit may identify the positions of multiple passersby, and the prediction unit may identify multiple candidate routes for each passerby to travel from the identified positions to the first area, and may obtain an entry indicator for each passerby based on the possibility that each candidate route will take each of the multiple candidate routes. In this case, it is possible to appropriately predict whether or not a passerby will enter the first area.

[8] In the behavior prediction device described in any one of [1] to [7] above, the aggregation unit may perform statistical processing on the entry index of each passerby to obtain a composite index. In this case, it is possible to appropriately predict whether at least one of the passersby will enter the first area.

[9] In the behavior prediction device described in any one of [1] to [8] above, the aggregation unit may obtain a composite index by a weighted average using the entry index of each passerby as a weighting factor. By taking a weighted average using the entry index of each passerby as a weighting factor, it is possible to predict with high accuracy whether at least one of the multiple passersby will enter the first area.

[10] In the behavior prediction device described in any one of [1] to [8] above, the aggregation unit may obtain an index for which all passersby do not enter the first area based on the entry index of each passerby, and may subtract the index for which all passersby do not enter the first area from the overall index to obtain a composite index. In this case, it is possible to predict with high accuracy whether at least one of multiple passersby will enter the first area.

[11] A behavior prediction method according to one aspect of the present disclosure includes the steps of: analyzing an image captured of an area including a second area surrounding a first area to detect multiple passersby appearing in the image; determining an entry index indicating the possibility that each passerby will enter the first area; and aggregating the entry indexes of each passerby to determine a composite index indicating the possibility that at least one of the multiple passersby will enter the first area. With this behavior prediction method, in a scene where multiple passersby are present, it is possible to predict whether or not at least one of the multiple passersby will enter the first area.

[Details of the embodiment of the present disclosure]
Specific examples of the embodiments of the present disclosure will be described below with reference to the drawings. The present invention is not limited to these examples, but is indicated by the claims, and is intended to include all modifications within the meaning and scope of the claims. In the description of the drawings, the same elements are given the same reference numerals, and duplicate descriptions are omitted.

FIG. 1 is a schematic diagram of a behavior prediction system 1 according to one embodiment. The behavior prediction system 1 predicts the behavior of multiple passersby M, in particular, whether or not at least one of the multiple passersby M will cross a pedestrian crossing.

In the following explanation, the behavior prediction system 1 is applied to a scene in which a sidewalk 2 and a roadway 3 are provided, and an example will be explained in which the system predicts whether at least one of a plurality of pedestrians M walking on the sidewalk 2 will cross a crosswalk 4 installed on the roadway 3. The pedestrians M are typically pedestrians, but pedestrians M also include people riding on vehicles such as bicycles, kickboards, and wheelchairs. As shown in FIG. 1, pedestrian signals S1 are installed on both sides of the crosswalk 4 in the direction of travel. The roadway 3 is equipped with vehicle signals S2 for vehicles passing through the crosswalk 4. The sidewalk 2 may be a shoulder or a shoulder strip. The roadway 3 is a road that pedestrians M may cross. In the following explanation, among the pedestrians M walking on the sidewalk 2, pedestrians M who intend to cross the crosswalk 4 may be referred to as "crossers".

As shown in FIG. 1, the behavior prediction system 1 includes a camera 10, a roadside device 20, and a behavior prediction device 30. The camera 10 is an imaging device that is installed, for example, on the top of a pole 5 installed on the sidewalk 2 and captures an image of an imaging area C. In the following description, "image" includes moving images and still images. Although not limited to this, the installation height of the camera 10 may be 3 m or more from the ground surface. In one embodiment, as shown in FIG. 2, the imaging area C is an area that includes a waiting area (second area) R2 around the roadway 3. The waiting area R2 is located in front of the crosswalk 4, and is an area on the sidewalk 2 where a passerby M who is about to cross the crosswalk 4 temporarily waits. As shown in FIG. 2, the imaging area C may include a part or the entire crossing area (first area) R1 that includes the crosswalk 4.

The camera 10 may be disposed at an angle as long as it can capture an image of the imaging area C. For example, the central axis (optical axis) of the camera 10 may be perpendicular to the ground surface, or may be inclined relative to the ground surface. The camera 10 may be installed on the opposite side of the crosswalk 4 from the imaging area C.

The camera 10 captures the image capture area C at a predetermined frame rate and generates a moving image. The captured moving image is transmitted to the roadside device 20 via wired or wireless communication. The camera 10 may also periodically transmit images (still images) captured in the image capture area C to the roadside device 20.

The roadside device 20 is a computer such as a PLC (Programmable Logic Controller) equipped with a processor, a storage device, a communication device, etc., and is installed adjacent to the roadway 3. For example, the roadside device 20 is provided on a support pole 5. The roadside device 20, for example, loads a program stored in a storage device and executes the loaded program with a processor to realize various functions described below.

The roadside device 20 is connected to the camera 10, the behavior prediction device 30, the pedestrian traffic light S1, and the vehicle traffic light S2 so that they can communicate with each other. The roadside device 20 transmits video images captured by the camera 10 to the behavior prediction device 30. The roadside device 20 also receives control signals from the behavior prediction device 30 and controls the pedestrian traffic light S1 and the vehicle traffic light S2.

The behavior prediction device 30 is a stationary or portable computer or workstation equipped with a processor, a storage device, a communication device, etc., and is typically located at a location away from the roadside device 20.

The behavior prediction device 30 can communicate with the roadside device 20 via the network N. The network N is a communication network for wireless or wired communication. As described below, the behavior prediction device 30 obtains an index indicating the possibility that any one of multiple passersby M captured in a video image captured by the camera 10 will enter the crossing area R1. The index is, for example, a probability, likelihood, or reliability. Note that "entering the crossing area R1" is a concept that includes the passerby M passing through the boundary between the crossing area R1 and the waiting area R2. The behavior prediction device 30, for example, loads a program stored in a storage device and executes the loaded program on a processor to realize various functions described below.

As shown in FIG. 3, the behavior prediction device 30 has, as its functional configuration, an acquisition unit 31, a detection unit 32, a prediction unit 33, an aggregation unit 34, and a control unit 35. The acquisition unit 31 acquires video images of the imaging area C captured by the camera 10 via the roadside device 20. The acquisition unit 31 acquires video images captured by the camera 10 that show at least one passerby M. The acquisition unit 31 may periodically acquire images (still images) that show multiple passersby M.

The detection unit 32 analyzes the video acquired by the acquisition unit 31 and detects multiple passersby M appearing in the video. The detection unit 32 identifies multiple passersby M appearing in each frame (image) of the video, for example by image recognition processing, and outputs position information indicating the positions of the identified multiple passersby M. Figure 4(a) shows an example of one frame included in the video input to the detection unit 32. Figure 4(b) shows an example of a frame including position information of passersby M. In Figures 4(a) and 4(b), passersby M1, passersby M2, and passersby M3 are shown as multiple passersby M.

The position information of passersby M can be identified using known image recognition algorithms such as Convolutional Neural Network (CNN), Histogram of Oriented Gradients (HOG), and You Only Look Once (YOLO). In addition to the position information of each passerby M, the detection unit 32 may further identify attribute information indicating the attributes of each passerby M. The attribute information of passersby M is information indicating the attributes of passersby M, such as the age, gender, and type of passerby M (pedestrian, bicycle, wheelchair, kickboard, white cane holder, etc.). The detection unit 32 identifies the position information and attributes of each passerby M for all frames included in the video.

The detection unit 32 may also identify the movement trajectory of the passerby M based on the position information of the passerby M in multiple frames. For example, the detection unit 32 connects the positions of each passerby M between multiple frames arranged in chronological order to generate the movement trajectory of each passerby M. The movement trajectory of the passerby M can be generated using known tracking algorithms such as a Kalman filter, a particle filter, Optical Flow, and Deep SORT.

Figure 5 shows an example of the movement trajectory of each passerby M identified by the detection unit 32. Movement trajectory 50A in Figure 5 represents the movement trajectory of passerby M1, movement trajectory 50B represents the movement trajectory of passerby M2, and movement trajectory 50C represents the movement trajectory of passerby M3. The detection unit 32 associates information indicating the movement trajectory of each passerby M with an identifier (ID) that identifies each passerby M, and outputs the information to the prediction unit 33.

The prediction unit 33 calculates an entry index for each pedestrian M, which indicates the possibility that each pedestrian M will enter the crossing area R1. The entry index is the probability, likelihood, or reliability that each pedestrian M will enter the crossing area R1. In the following explanation, an example of calculating an entry probability, which indicates the probability that each pedestrian M will enter the crossing area R1, as an entry index, will be explained. The entry probability can also be said to be the probability that each pedestrian M will cross the crosswalk 4. The entry probability for each pedestrian M can be calculated using various methods.

A first method for calculating the entry probability of each passerby M will be described. In the first method, the entry probability of each passerby M is calculated using a prediction model that has been machine-learned to input the movement trajectory of each passerby M and output the entry probability of each passerby M. The prediction model is a classifier constructed by machine learning a data set of teacher data including the movement trajectory of passerby M captured in a video image captured in the past and label information indicating the presence or absence of a crossing person. The prediction model is generated by optimizing learning parameters using known machine learning algorithms such as a convolutional neural network and a recurrent neural network. The prediction unit 33 reads out the prediction model stored in the storage device, inputs the movement trajectory of each passerby M identified by the detection unit 32 into the prediction model, and obtains the probability that each passerby M will enter the crossing area R1.

The training data used to generate the prediction model may include attribute information indicating the attributes of passersby M. The behavior of passersby M may have characteristic patterns depending on their attributes. For example, adult pedestrians tend to move quickly and in a straight line. In contrast, the movement speeds of children, pedestrians with strollers, and people using white canes tend to be slower than those of adult pedestrians. Passersby M riding bicycles or kickboards tend to move quickly and change their movement direction less compared to pedestrians. When the training data includes attribute information of passersby M, a prediction model can be generated that takes into account the movement patterns of passersby M described above.

As described above, when a prediction model is generated using training data including attribute information of passersby M, the prediction unit 33 inputs the attribute information of passersby M identified by the detection unit 32 into the prediction model in addition to the movement trajectory of each passerby M. This makes it possible to calculate the probability of entry of each passerby M with high accuracy.

Next, a second method for calculating the entry probability of each passerby M will be described. In the second method, the entry probability of each passerby M is calculated based on the distance between each passerby M and the crossing area R1. In general, it is considered that the closer the position of passerby M is to the crossing area R1, the more likely the passerby M will enter the crossing area R1. Therefore, in the second method, the prediction unit 33 calculates the distance between each passerby M and the crossing area R1 based on the position information of each passerby M identified by the detection unit 32, and sets the entry probability of passerby M higher the shorter the distance.

The prediction unit 33 may also determine the movement direction of each passerby M from the movement trajectory of each passerby M, and set the entry probability of the passerby M to be high when the passerby M is moving toward the crossing area R1, and set the entry probability of the passerby M to be low when the passerby M is moving in a direction away from the crossing area R1.

In addition, even if the distance between passerby M and crossing area R1 is small, if the movement speed of passerby M is fast, there is a high possibility that passerby M will enter crossing area R1. Therefore, the prediction unit 33 may set the entry probability of passerby M high if the movement speed of passerby M is fast, and set the entry probability of passerby M low if the movement speed of passerby M is slow. The movement speed of passerby M is determined based on the attributes of passerby M, for example.

With reference to FIG. 6, an example of a method for determining the entry probability of each passerby M using the second method will be described. In FIG. 6, passerby M1, passerby M2, passerby M3, and passerby M4 are illustrated as multiple passersby M. Passersby M1, M2, and M4 are pedestrians, and passerby M3 is a person riding a bicycle. In the scene shown in FIG. 6, passerby M1 is located closer to the crossing area R1 than passersby M2, M3, and M4, so the prediction unit 33 sets the entry probability of passerby M1 higher than the entry probabilities of passersby M2, M3, and M4. Passerby M3 is riding a bicycle and moves faster than passersby M1, M2, and M3, so the prediction unit 33 sets, for example, the entry probability of passerby M3 higher than the entry probability of passerby M2.

Next, a third method for determining the entry probability of each passerby M will be described. In the third method, multiple candidate routes for moving from the position of each passerby M to the crossing area R1 are identified, and the entry probability of each passerby M is determined based on the probability that each passerby M will take each of the multiple candidate routes. With reference to FIG. 7, an example of a method for determining the entry probability of each passerby M using the third method will be described. FIG. 7 illustrates a passerby M located in the imaging area C. In the third method, the prediction unit 33 first sets multiple candidate routes 51A, 51B, and 51C for moving from the position of the passerby M to the crossing area R1. Note that the number of candidate routes set by the prediction unit 33 is arbitrary.

Next, the prediction unit 33 determines the probability that passerby M will select multiple route candidates 51A, 51B, and 51C as the route along which passerby M will travel. The probability that passerby M will select route candidates 51A, 51B, and 51C can be determined, for example, based on the past movement trajectory of passerby M. In the following description, the probability that passerby M will select route candidate 51A is represented as P1(A), the probability that passerby M will select route candidate 51B is represented as P1(B), and the probability that passerby M will select route candidate 51C is represented as P1(C).

Next, the prediction unit 33 determines the probability that passerby M will travel along the candidate routes 51A, 51B, and 51C. The probability that passerby M will travel along the candidate routes 51A, 51B, and 51C indicates the probability that passerby M will travel from the start point to the end point of each of the candidate routes 51A, 51B, and 51C without turning back along the way. In the following description, the probability that passerby M will travel along candidate route 51A is represented as P2(A), the probability that passerby M will travel along candidate route 51B is represented as P2(B), and the probability that passerby M will travel along candidate route 51C is represented as P2(C). Probabilities P2(A) to P2(C) can be determined based on, for example, the past movement trajectories of passerby M.

Next, the prediction unit 33 calculates the probability that the passerby M will enter the traversal area R1 via the route candidates 51A, 51B, and 51C. In the following description, the probability that the passerby M will enter the traversal area R1 via the route candidate 51A is represented as P3(A), the probability that the passerby M will enter the traversal area R1 via the route candidate 51B is represented as P3(B), and the probability that the passerby M will enter the traversal area R1 via the route candidate 51C is represented as P3(C). At this time, the probabilities P3(A), P3(B), and P3(C) are calculated by the following formulas.
P3(A)=P1(A)・P2(A)
P3(B)=P1(B)・P2(B)
P3(C)=P1(C)・P2(C)

Next, the prediction unit 33 calculates the probability P(M) of the pedestrian M entering the crossing area R1 using the following formula.
P(M)=P3(A)+P3(B)+P3(C)

The prediction unit 33 outputs the entry probability of each passerby M calculated using the various methods described above to the aggregation unit 34.

The aggregation unit 34 aggregates the entry probabilities of each passerby M to obtain a composite index indicating the possibility that at least one of the multiple passersby M will enter the crossing area R1. The composite index is the probability, likelihood, or reliability that at least one of the multiple passersby M will enter the crossing area R1. In the following explanation, an example of obtaining a composite probability indicating the probability that at least one of the multiple passersby M will enter the crossing area R1 as the composite index will be explained. The composite probability can also be said to be the probability that at least one of the multiple passersby M will cross the crosswalk 4. For example, the aggregation unit 34 performs statistical processing on the entry probabilities of each passerby M obtained by the prediction unit 33 to obtain the composite probability.

The composite probability can be calculated by various methods. First, a first method for calculating the composite probability will be described. In the first method, the aggregation unit 34 calculates the composite probability by averaging or weighted averaging with the entry probability of each passerby M as a weighting factor. For example, if the entry probability of a plurality of passersby Mn (n=1, 2, 3, . . . , N) into the crossing area R1 is represented as P(Mn), the aggregation unit 34 can calculate the composite probability Pc by the following formula:
Pc=(P(M1)+P(M2)+P(M3)+...+P(MN))/N

As described above, the composite probability Pc is calculated by dividing the sum of the entry probabilities of multiple passersby M by the number of passersby M. For example, if there are 10 passersby M in the video and the entry probabilities of the 10 passersby are 0.9, 0.8, 0.7, 0.9, 0.6, 0.5, 0.8, 0.9, 0.4, and 0.7, respectively, the composite probability Pc is 0.72. This result indicates that the probability that at least one of the 10 passersby will cross the crosswalk 4 is 72%.

Next, a second method for calculating the composite probability will be described. In the second method, the aggregation unit 34 uses the complement to subtract the probability that none of the passersby M enter the crossing area R1 from the overall probability to calculate the composite probability Pc. For example, if the probability of multiple passersby Mn (n=1, 2, 3..., N) entering the crossing area R1 is expressed as P(Mn), the aggregation unit 34 calculates the probability Pf that none of the multiple passersby Mn enters the crossing area R1 using the following formula.
Pf=(1-P(M1))・(1-P(M2))・(1-P(M3))...(1-P(MN))

Then, the aggregation unit 34 calculates the composite probability Pc by the following formula.
Pc=1-Pf

As described above, in the second method, the combined probability Pc is calculated by subtracting the probability Pf that none of the pedestrians M will enter the crossing area R1 from the overall probability. For example, if the number of pedestrians M captured in the video is four, and the entry probabilities of the four pedestrians are 0.7, 0.3, 0.6, and 0.4, respectively, the combined probability Pc is 0.9496. This result indicates that the probability that at least one of the four pedestrians will cross the crosswalk 4 is approximately 95%.

In addition, the aggregation unit 34 can determine the composite probability by performing any statistical processing on the entry probability of each passerby M using a method other than the above. For example, the aggregation unit 34 can determine the composite probability by performing a likelihood evaluation, or can determine the composite probability using a probability mass function. The aggregation unit 34 outputs the determined composite probability to the control unit 35.

The control unit 35 controls the pedestrian traffic light S1 and the vehicular traffic light S2 based on the composite probability output from the aggregation unit 34. For example, if the composite probability is higher than a reference value, the control unit 35 controls the pedestrian traffic light S1 and the vehicular traffic light S2 via the roadside device 20 so that the pedestrian traffic light S1 turns green and the vehicular traffic light S2 turns red. If the pedestrian traffic light S1 is a push-button type traffic light that changes the signal display by pressing a button, the control unit 27 may output a control signal to physically or electrically press the button. This allows pedestrians M to cross the crosswalk 4 safely. The reference value is a setting value that is preset by the designer.

Next, the hardware configuration of the behavior prediction device 30 will be described. FIG. 8 is a block diagram showing an example of the hardware configuration of the behavior prediction device 30. The behavior prediction device 30 is composed of one or more computers. The computer includes a CPU (processor) 101, a main memory unit 102, an auxiliary memory unit 103, a communication control unit 104, an input device 105, and an output device 106. Each of the behavior prediction devices 30 is composed of one or more computers that are composed of this hardware and software such as programs.

When the behavior prediction device 30 is composed of multiple computers, the multiple computers may be connected locally or via a communication network such as the Internet or an intranet. This connection logically constructs a single behavior prediction device 30.

The CPU 101 is a processor that executes an operating system, application programs, etc. The main memory unit 102 is composed of a ROM (Read Only Memory) and a RAM (Random Access Memory). The auxiliary memory unit 103 is a storage medium composed of a hard disk, a flash memory, etc. The auxiliary memory unit 103 generally stores a larger amount of data than the main memory unit 102. The communication control unit 104 is composed of a network card or a wireless communication module. At least a part of the communication function between the camera 10, the roadside device 20, and the behavior prediction device 30 is realized by the communication control unit 104. The input device 105 is composed of a keyboard, a mouse, a touch panel, a microphone for voice input, etc. The output device 106 is composed of a display, a printer, etc.

The auxiliary memory unit 103 stores programs and data necessary for processing. The programs cause the computer to execute each functional element of the behavior prediction device 30. The functions of the behavior prediction device 30 are realized in the computer by the programs. For example, the programs are loaded by the CPU 101 or the main memory unit 102, and cause at least one of the CPU 101, the main memory unit 102, the auxiliary memory unit 103, the communication control unit 104, the input device 105, and the output device 106 to operate. For example, the programs read and write data in the main memory unit 102 and the auxiliary memory unit 103.

The program may be provided in a form recorded on a tangible storage medium such as a CD-ROM, DVD-ROM, or semiconductor memory. The program may be provided as a data signal via a communications network.

The behavior prediction device 30 does not necessarily have to be configured as a computer that operates according to a program, and some or all of the functions of the behavior prediction device 30 may be implemented in an ASIC (Application Specific Integrated Circuit) that integrates logic circuits.

Next, a behavior prediction method according to one embodiment will be described. FIG. 9 is a flowchart showing a behavior prediction method according to one embodiment. This behavior prediction method is executed by the behavior prediction device 30 described above.

As shown in FIG. 9, first, the acquisition unit 31 of the behavior prediction device 30 acquires an image of the imaging area C captured by the camera 10 (step ST1). The image acquired by the acquisition unit 31 is an image captured by the camera 10 that contains at least one passerby M. The acquisition unit 31 periodically acquires images from the camera 10, for example.

Next, the detection unit 32 detects multiple passersby M appearing in the image (step ST2). At this time, the detection unit 32 acquires position information and attribute information of each passerby M. Next, the detection unit 32 tracks each passerby M based on the time-series position information of each passerby M to identify the movement trajectory (step ST3).

Next, the prediction unit 33 calculates the probability of entry for each passerby M (step ST4). For example, the prediction unit 33 inputs the movement trajectory of each passerby M into the above-mentioned prediction model, and calculates the probability that multiple passersby M will enter the crossing area R1 for each passerby M.

Next, the prediction unit 33 aggregates the entry probabilities of each passerby M to obtain a composite probability indicating the probability that at least one of the multiple passersby M will enter the crossing area R1 (step ST5). For example, the prediction unit 33 performs statistical processing on the entry probabilities of each passerby M, such as a weighted average in which the entry probabilities are weighted, to obtain the composite probability.

Next, the control unit 35 determines whether the combined probability of multiple passersby M output from the aggregation unit 34 is higher than a reference value (step ST6). If the combined probability is higher than the reference value, the control unit 35 sends a control signal to the roadside device 20 to control the pedestrian traffic light S1 and the vehicular traffic light S2 so that the passerby M can cross the crosswalk 4 (step ST7). On the other hand, if the crossing probability is equal to or lower than the reference value, the control unit 35 ends the series of processes without controlling the pedestrian traffic light S1 and the vehicular traffic light S2.

As described above, the behavior prediction device 30 aggregates the probability that each passerby M will enter the crossing area R1 to determine the probability that any one of the multiple passersby M will cross the crossing area R1, so that in a scene where multiple passersby M are present, it is possible to predict whether or not at least one of the multiple passersby M will enter the first area.

The above describes the behavior prediction device 30 and behavior prediction method according to various embodiments, but the invention is not limited to the above-described embodiments and can be modified in various ways without departing from the spirit of the invention.

For example, in the above embodiment, the behavior prediction device 30 acquires images from the camera 10 via the roadside device 20 and controls the pedestrian traffic light S1 and the vehicle traffic light S2 via the roadside device 20, but the behavior prediction device 30 may acquire images directly from the camera 10 and directly control the pedestrian traffic light S1 and the vehicle traffic light S2. The behavior prediction device 30 may be integrated with the camera 10. That is, the behavior prediction device 30 may be provided inside the housing of the camera 10. The behavior prediction device 30 may be integrated with the roadside device 20. That is, the behavior prediction device 30 may be provided inside the housing of the roadside device 20.

In addition, in the above embodiment, whether each passerby M will enter the crossing area R1 is predicted based on the movement trajectory of the passerby M, but it is possible to determine whether the passerby M will enter the crossing area R1 using any method. For example, the prediction unit 33 may determine the entry probability of each passerby M using only the position information of each passerby M without using the movement trajectory of each passerby.

In the above embodiment, the behavior prediction device 30 predicts whether at least one of multiple pedestrians M will cross the crosswalk 4, but the behavior prediction device 30 can also be applied to roads where the crosswalk 4 is not installed. For example, the behavior prediction device 30 can predict whether at least one of multiple pedestrians M shown in an image will enter a specific area on the road. Also, it is not necessary for the pedestrian traffic light S1 and the vehicular traffic light S2 to be installed, and the behavior prediction device 30 only needs to predict the entry of a pedestrian M into a specific area, and does not necessarily have to control the pedestrian traffic light S1 and the vehicular traffic light S2.

The various embodiments described above can be combined to the extent that no contradictions arise.

1... Behavior prediction system 2... Sidewalk 3... Roadway 4... Crosswalk 5... Support pole 10... Camera 20... Roadside device 30... Behavior prediction device 31... Acquisition unit 32... Detection unit 33... Prediction unit 34... Aggregation unit 35... Control unit 50A, 50B, 50C... Movement trajectory 51A, 51B, 51C... Route candidate 101... CPU (processor)
102: Main memory unit 103: Auxiliary memory unit 104: Communication control unit 105: Input device 106: Output device C: Image capture area M: Passersby N: Network R1: Crossing area (first area)
R2: Waiting area (second area)
S1: Pedestrian traffic light S2: Vehicle traffic light

Claims (11)

a detection unit that analyzes an image captured of an area including a second area around the first area and detects a plurality of passersby appearing in the image;
a prediction unit for determining an entry indicator indicating a possibility that each passerby will enter the first area;
an aggregation unit that aggregates the entry indicators of each of the plurality of passersby to obtain a composite indicator indicating a possibility that at least one of the plurality of passersby will enter the first area;
A behavior prediction device comprising: The first area is an area including a pedestrian crossing where a traffic light is installed,
The behavior prediction device according to claim 1 , further comprising a control unit configured to control the traffic light so that the plurality of pedestrians can cross the crosswalk when the composite index is higher than a reference value. The detection unit identifies a movement trajectory of each of the plurality of passersby based on position information of the passersby;
The behavior prediction device according to claim 1 , wherein the prediction unit determines the entry index of each passerby based on the movement trajectory. The behavior prediction device according to claim 3, wherein the prediction unit inputs the movement trajectory of each identified passerby and obtains the entry index of each passerby using a prediction model that has been machine-learned to output the entry index of each passerby. The detection unit identifies positions of the plurality of passersby,
The behavior prediction device according to claim 1 , wherein the prediction unit determines the entry index of each passerby based on a distance between each passerby and the first area. The prediction unit further identifies attributes of the plurality of passersby;
The behavior prediction device according to claim 5 , wherein the prediction unit determines the entry index of each passerby based on the distance and an attribute of each passerby. The detection unit identifies positions of the plurality of passersby,
3. The behavior prediction device according to claim 1, wherein the prediction unit identifies a plurality of candidate routes for each passerby to travel from the identified position to the first area, and calculates the entry index of each passerby based on a possibility that each candidate route will pass through each of the plurality of candidate routes. The behavior prediction device according to any one of claims 1 to 7, wherein the aggregation unit performs statistical processing on the entry index of each passerby to obtain the composite index. The behavior prediction device according to any one of claims 1 to 8, wherein the aggregation unit calculates the composite index by a weighted average in which the entry index of each passerby is weighted. The behavior prediction device according to any one of claims 1 to 8, wherein the aggregation unit determines an index indicating that all passersby do not enter the first area based on the entry index of each passerby, and subtracts the index indicating that all passersby do not enter the first area from the overall index to obtain the composite index. A step of analyzing an image captured in an area including the second area around the first area and detecting a plurality of passersby captured in the image;
determining an entry indicator indicative of a possibility that each passerby will enter the first area;
aggregating the entry indicators of each passerby to obtain a composite indicator indicative of a likelihood that at least one of the plurality of passersby will enter the first area;
A behavior prediction method comprising:

Images