WO2025181991A1 - Information processing device, information processing method, and program - Google Patents

Abstract

In order to improve the accuracy of detecting a facility from an image obtained by imaging facilities around a vehicle in various environments, an information processing device comprises: an acquisition unit that acquires an image of the surrounding of the vehicle; an estimation unit that refers to the image of the surrounding of the vehicle and estimates an environment at the time of imaging; and a detection unit that, as a reference for detecting a facility included as a subject from the image of the surroundings of the vehicle, detects the facility using a progressively lower reference as the difficulty level of detection that corresponds to the estimated environment increases.

Description

Information processing device, information processing method, and program

The present invention relates to an information processing device, information processing method, and program that uses images of facilities around a vehicle.

Technology is known that detects equipment around a vehicle from images of the equipment.

Patent Document 1 discloses an overhead line fitting detection device that detects overhead line fittings using line sensor images output from two line sensor cameras installed on a railway vehicle. The line sensor image is generated by combining line images captured continuously in a line while the line sensor or the object is moving. The overhead line fitting detection device performs day/night determination processing on each line image that makes up the line sensor image. Furthermore, the overhead line fitting detection device detects overhead wires and overhead line fittings by inverting the brightness of areas where a certain number of consecutive lines have a brightness value above a threshold for line images determined to be night.

Japanese Patent Application Publication No. 2020-149286

The overhead line fitting detection device described in Patent Document 1 can only detect overhead line fittings if the area around the overhead line fitting is bright in the line sensor image. As a result, it is unable to detect overhead line fittings from line sensor images taken in environments where the area around the overhead line fitting is not bright (for example, inside a tunnel, under a bridge or road). In other words, the overhead line fitting detection device has the problem of being limited in the environments in which images for detecting overhead line fittings can be taken. Therefore, there is a demand for technology that can detect equipment included as subjects in images taken in a variety of environments.

One aspect of the present invention was made in consideration of the above-mentioned problems, and one of its objectives is to provide technology that improves the accuracy of detecting facilities around a vehicle from images of the facilities taken in various environments.

An information processing device according to one aspect of the present invention comprises an acquisition means for acquiring a vehicle surroundings image, an estimation means for estimating the environment at the time of photographing by referring to the vehicle surroundings image, and a detection means for detecting equipment included as a subject in the vehicle surroundings image using lower criteria as the difficulty of detection increases according to the estimated environment.

An information processing method according to one aspect of the present invention includes an information processing device acquiring a vehicle surroundings image, estimating the environment at the time of photographing by referring to the vehicle surroundings image, and detecting equipment included as a subject in the vehicle surroundings image using lower criteria as the difficulty of detection increases according to the estimated environment.

A program according to one aspect of the present invention causes a computer to function as an information processing device, and causes the computer to function as: an acquisition means for acquiring a vehicle surroundings image; an estimation means for estimating the environment at the time of photographing by referring to the vehicle surroundings image; and a detection means for detecting equipment included as a subject in the vehicle surroundings image using lower criteria as the difficulty of detection increases according to the estimated environment.

One aspect of the present invention makes it possible to improve the accuracy of detecting facilities around a vehicle from images of the facilities taken in various environments.

1 is a block diagram showing a configuration of an information processing device according to a first exemplary embodiment of the present invention; FIG. 1 is a flowchart showing the flow of an information processing method according to a first exemplary embodiment of the present invention. 10 is a table showing an example of equipment and status in exemplary embodiment 2 of the present invention. FIG. 10 is a schematic diagram showing an example of a railway vehicle TR and equipment in exemplary embodiment 2 of the present invention. FIG. 10 is a block diagram showing a configuration of an information processing device 2 according to an exemplary embodiment 2 of the present invention. FIG. 10 is a diagram illustrating a process for training an estimation model in exemplary embodiment 2 of the present invention. FIG. 10 is a flowchart showing the flow of an information processing method according to a second exemplary embodiment of the present invention. FIG. 10 is a diagram illustrating an example of processing by an estimation unit according to the second exemplary embodiment of the present invention. FIG. 2 is a block diagram showing an example of a hardware configuration of an information processing device according to each exemplary embodiment of the present invention.

[Example Embodiment 1]
A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described below.

(Configuration of information processing device 1)
The configuration of an information processing device 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of an information processing device 1 according to this exemplary embodiment.

As shown in FIG. 1, the information processing device 1 includes an acquisition unit 11, an estimation unit 12, and a detection unit 13. In this exemplary embodiment, the acquisition unit 11, the estimation unit 12, and the detection unit 13 are configured to realize an acquisition means, an estimation means, and a detection means, respectively.

The acquisition unit 11 acquires an image of the vehicle's surroundings. The acquisition unit 11 supplies the acquired image of the vehicle's surroundings to the estimation unit 12.

The estimation unit 12 estimates the environment at the time of shooting by referring to the vehicle surroundings image supplied from the acquisition unit 11. The estimation unit 12 supplies the vehicle surroundings image and the estimation result to the detection unit 13.

The detection unit 13 uses lower criteria to detect equipment included as a subject in the vehicle surroundings image supplied from the estimation unit 12, and detects the equipment as the detection difficulty increases according to the environment estimated by the estimation unit 12.

As described above, the information processing device 1 according to this exemplary embodiment is configured to include an acquisition unit 11 that acquires vehicle surroundings images, an estimation unit 12 that estimates the environment at the time of shooting by referring to the vehicle surroundings images supplied from the acquisition unit 11, and a detection unit 13 that detects equipment included as a subject from the vehicle surroundings images supplied from the estimation unit 12 using lower criteria that correspond to higher detection difficulty in accordance with the environment estimated by the estimation unit 12.

As a result, the information processing device 1 according to this exemplary embodiment has the effect of improving the accuracy of detecting facilities around a vehicle from images of the facilities taken in various environments.

(Flow of information processing method S1)
The flow of the information processing method S1 according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the information processing method S1 according to this exemplary embodiment.

(Step S11)
In step S11, the acquisition unit 11 acquires a vehicle surroundings image and supplies the acquired vehicle surroundings image to the estimation unit 12.

(Step S12)
In step S12, the estimation unit 12 estimates the environment at the time of shooting by referring to the vehicle surroundings image supplied from the acquisition unit 11. The estimation unit 12 supplies the vehicle surroundings image and the estimation result to the detection unit 13.

(Step S13)
In step S13, the detection unit 13 detects equipment included as a subject from the vehicle surroundings image supplied from the estimation unit 12 using a lower criterion as the difficulty of detection increases according to the environment estimated by the estimation unit 12.

As described above, the information processing method S1 according to this exemplary embodiment includes the following configuration: step S11 in which the acquisition unit 11 acquires a vehicle surroundings image; step S12 in which the estimation unit 12 estimates the environment at the time of image capture by referring to the vehicle surroundings image supplied from the acquisition unit 11; and step S13 in which the detection unit 13 detects equipment included as a subject from the vehicle surroundings image supplied from the estimation unit 12 using lower criteria as the difficulty of detection increases according to the environment estimated by the estimation unit 12. Therefore, the information processing method S1 according to this exemplary embodiment achieves the same effects as the information processing device 1 described above.

Exemplary Embodiment 2
A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are denoted by the same reference numerals, and their description will be omitted as appropriate.

(Overview of information processing device 2)
The information processing device 2 according to this exemplary embodiment is a device that analyzes the state of facilities around a vehicle. Although the vehicle is not particularly limited, this exemplary embodiment describes a case where the information processing device 2 analyzes the state of facilities around a railway vehicle.

Peripheral equipment for railway vehicles refers to equipment installed around railway vehicles in order to allow the vehicles to run. For example, the equipment is equipment used to supply power to railway vehicles. Examples of equipment include wires, contact wires, hangers, ears, connectors, bolts, and insulators.

Furthermore, the state of equipment is a state that can be determined from the appearance of the equipment. For example, the state of equipment may be either a good state or a bad state. Furthermore, what constitutes a good state and what constitutes a bad state may differ depending on the equipment, in which case the equipment and the state may be specified in association with each other.

An example of how equipment and status are associated and defined will be described with reference to Figure 3. Figure 3 is a table showing an example of equipment and status in this exemplary embodiment.

In the table shown in Figure 3, equipment and states are associated. For example, equipment "W Year" is associated with the states "CC Misalignment" and "Bolt Falling Off," and equipment "Protector" is associated with the state "Bolt Falling Off."

Based on the table in Figure 3, the information processing device 2 analyzes whether the state of the equipment is in a state associated with that equipment. As one example, the information processing device 2 analyzes whether the equipment "W Year" is in the state "CC Misaligned" and whether the equipment "W Year" is in the state "Bolt Missing." As another example, the information processing device 2 analyzes whether the equipment "Protector" is in the state "Bolt Missing."

Note that the status in the table in Figure 3 is a status in which a malfunction has occurred in the equipment. Therefore, it can also be said that the information processing device 2 analyzes whether or not the equipment is in a defective state.

Furthermore, in order to analyze the condition of the equipment, the information processing device 2 acquires vehicle surroundings images, which are images captured by cameras installed on surfaces exposed to the outside of the railway vehicle and which include multiple pieces of equipment as subjects. The vehicle surroundings images may be images captured while the vehicle is in motion. The surfaces exposed to the outside include, for example, the top surface, bottom surface, left and right sides, and some or all of the front and rear sides of the railway vehicle. In this exemplary embodiment, the surface on which the camera is installed will be described mainly as the top surface of the railway vehicle, but is not limited to this. Hereinafter, "top surface of the railway vehicle" will also be simply referred to as "on the railway vehicle." The cameras installed on the railway vehicle and the equipment that serves as the subject will be described with reference to Figure 4. Figure 4 is a schematic diagram showing an example of a railway vehicle TR and equipment in this exemplary embodiment.

As shown in Figure 4, multiple cameras (cameras CA1 to CA6) are installed on the railway vehicle TR, capturing images of equipment within their angle of view and outputting the captured images. It is desirable that at least a portion of the ranges within the angle of view of the multiple cameras are different from each other. In other words, the range within the angle of view of each camera (hereinafter also referred to as the "capture range") may partially overlap with the range within the angle of view of at least one other camera, but it is desirable that at least a portion of the ranges are different.

The installation manner of the multiple cameras is not particularly limited, but as an example, as shown in Figure 4, three cameras CA1 to CA3 are installed at different heights on the right side of the vehicle TR when viewing the front of the vehicle TR from the direction of travel of the vehicle TR. Similarly, three cameras CA4 to CA6 are installed at different heights on the left side of the vehicle TR when viewing the front of the vehicle TR from the direction of travel of the vehicle TR.

Furthermore, there are no particular limitations on the timing at which cameras CA1 to CA6 photograph the equipment. One example is a configuration in which cameras CA1 to CA6 each photograph the equipment at a predetermined interval. Another example is a configuration in which multiple cameras photograph the equipment synchronously at predetermined intervals. One example of such a configuration is a configuration in which cameras CA1 and CA4 photograph the equipment synchronously, cameras CA2 and CA5 photograph the equipment synchronously, and cameras CA3 and CA6 photograph the equipment synchronously. However, images photographed by multiple cameras synchronously are not necessarily taken at exactly the same time due to processing delays, etc. Also, as one example, as shown in Figure 4, cameras CA1 to CA6 photograph the equipment, which are hangers HA, ears EA, and trolley wire TW, as their subjects.

Information processing device 2 acquires vehicle surroundings images captured by cameras CA1 to CA6. The configuration in which information processing device 2 acquires vehicle surroundings images is not particularly limited. One example is a configuration in which vehicle surroundings images captured by cameras CA1 to CA6 are stored on a recording medium, and information processing device 2 acquires vehicle surroundings images from the recording medium. Another configuration is a configuration in which information processing device 2 and cameras CA1 to CA6 are connected to each other so as to be able to communicate via a network, and information processing device 2 acquires vehicle surroundings images via the network.

(Configuration of information processing device 2)
The configuration of the information processing device 2 will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the configuration of the information processing device 2 according to this exemplary embodiment.

As shown in FIG. 5, the information processing device 2 includes a control unit 20, an input/output unit 27, a communication unit 28, and a memory unit 29.

The input/output unit 27 is an interface that accepts user input and outputs data. For example, the input/output unit 27 supplies information indicating the accepted user input to the control unit 20, and outputs information supplied from the control unit 20. Examples of the input/output unit 27 include, but are not limited to, a keyboard, a mouse, a touchpad, a microphone, and an LCD display.

The communication unit 28 is an interface that transmits and receives data over a network. For example, the communication unit 28 transmits data supplied from the control unit 20 to other devices, and supplies data received from other devices to the control unit 20. Examples of the communication unit 28 include, but are not limited to, communication chips for various communication standards such as Ethernet (registered trademark), Wi-Fi (registered trademark), and wireless communication standards for mobile data communication networks, as well as USB-compliant connectors.

The memory unit 29 stores data referenced by the control unit 20. Examples of data stored in the memory unit 29 include, but are not limited to, images of the vehicle's surroundings, images of equipment, and analysis results. Examples of the memory unit 29 include, but are not limited to, flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination of these.

(Functions of the control unit 20)
The control unit 20 controls each component included in the information processing device 2. As shown in Fig. 5 , the control unit 20 also includes an acquisition unit 11, an estimation unit 12, a detection unit 13, an image connection unit 21, and an analysis unit 22. In this exemplary embodiment, the acquisition unit 11, the estimation unit 12, the detection unit 13, and the analysis unit 22 are components that respectively realize an acquisition means, an estimation means, a detection means, and an analysis means.

(Acquisition unit 11)
The acquisition unit 11 acquires data supplied from the input/output unit 27 or the communication unit 28. The acquisition unit 11 also acquires data stored in the storage unit 29. As an example, the acquisition unit 11 acquires vehicle surroundings images from a plurality of cameras, cameras CA1 to CA6, via the input/output unit 27 or the communication unit 28. As another example, the acquisition unit 11 may acquire vehicle surroundings images captured by cameras CA1 to CA6 and stored in the storage unit 29.

(Estimation unit 12)
The estimation unit 12 estimates the environment at the time of image capture by referring to the vehicle surroundings image. The environment at the time of image capture estimated by the estimation unit 12 is not limited, but as an example, the estimation unit 12 estimates whether the environment at the time of image capture is a predetermined environment. Examples of the predetermined environment include "inside a tunnel,""under a bridge,""under a road," and "raining." As another example, the estimation unit 12 is configured to estimate which of two or more environments (e.g., inside a tunnel, under a bridge, and others) the environment at the time of image capture is. In this exemplary embodiment, the estimation unit 12 estimates whether the environment at the time of image capture is inside a tunnel. The estimation unit 12 supplies the vehicle surroundings image and the estimation result to the detection unit 13.

The method by which the estimation unit 12 estimates whether the environment at the time the vehicle surroundings image was captured is inside a tunnel is not limited. As an example, the estimation unit 12 may estimate whether the environment at the time the vehicle surroundings image was captured is inside a tunnel using an estimation model learned by machine learning.

The configuration for training the estimation model will be described with reference to FIG. 6. FIG. 6 is a diagram showing the process for training the estimation model MD1 in this exemplary embodiment. The estimation model receives an image of the vehicle's surroundings as input and outputs an estimation result indicating whether the image of the vehicle's surroundings was taken inside a tunnel. The learning process shown in FIG. 6 may be executed by another information processing device, or by the control unit 20 of the information processing device 2.

As shown in Figure 6, first, a vehicle surroundings image vp is associated with environmental information ei, which indicates whether the environment when the vehicle surroundings image vp was captured was inside a tunnel. In other words, the environmental information ei is the correct label. Next, the vehicle surroundings image vp is cropped, the image size is reduced, and contrast correction is performed to match the contrast between multiple vehicle surroundings images vp (for example, by performing CLAHE (Contrast Limited Adaptive Histogram Equalization)), generating a processed vehicle surroundings image p_vp. An estimation model MD1 is then trained using a training dataset that pairs the processed vehicle surroundings image p_vp with the environmental information ei. Specifically, when a vehicle surroundings image p_vp is input to the estimation model MD1, training is performed so that environmental information ei, which is the correct label and indicates whether the vehicle surroundings image p_vp was captured inside a tunnel, is output.

Another method by which the estimation unit 12 estimates whether the environment at the time of image capture is inside a tunnel is to refer to pixel values of the vehicle surroundings image. As an example, if the average brightness value of each pixel in the vehicle surroundings image is higher than a predetermined value, the estimation unit 12 estimates that the vehicle surroundings image was captured outside a tunnel, and if the average brightness value of each pixel in the vehicle surroundings image is equal to or lower than a predetermined value, the estimation unit 12 estimates that the vehicle surroundings image was captured inside a tunnel. However, the method by which the estimation unit 12 estimates the environment at the time of image capture is not limited to the above-mentioned method.

The estimation unit 12 may also divide the vehicle surroundings image into multiple partial images along a direction corresponding to the vehicle's traveling direction, and estimate whether the environment at the time of capture was inside a tunnel for each partial image after division. The direction corresponding to the vehicle's traveling direction may be the vehicle's forward direction. Specific examples of processing performed by the estimation unit 12 for this configuration will be described later.

(Detection unit 13)
The detection unit 13 detects equipment as a subject from the vehicle surroundings image using a lower criterion as the detection difficulty increases according to the environment indicated by the estimation result supplied from the estimation unit 12. The detection unit 13 supplies the estimation result and an equipment image including the detected equipment to the analysis unit 22. One equipment image includes one or more pieces of equipment as subjects.

High detection difficulty refers to a situation in which the equipment is difficult to distinguish from its surroundings in the vehicle surroundings image. One example is a situation in which the vehicle surroundings image is taken in dark conditions, and an object similar to the equipment is captured in the background of the equipment. For example, an image of the vehicle surroundings taken inside a tunnel may make the equipment difficult to see due to darkness, or the inner wall of the tunnel captured in the background of the equipment may resemble the equipment, making detection by the detection unit 13 more difficult.

Furthermore, low standards refer to lenient conditions for satisfying the standards. In other words, the more difficult it is to detect equipment, the more lenient the conditions under which the equipment is more easily detected by the detection unit 13. On the other hand, the less difficult it is to detect equipment, the more strict the conditions under which the equipment is more difficult to detect by the detection unit 13.

With this configuration, for equipment included as a subject in vehicle surroundings images taken in environments where detection is low, the detection unit 13 sets high standards for detecting the equipment, thereby reducing overdetection, in which subjects that are not equipment are detected as equipment. On the other hand, for equipment included as a subject in vehicle surroundings images taken in environments where detection is high, the detection unit 13 sets low standards for detecting the equipment, thereby allowing overdetection while reducing overlooking of equipment. Furthermore, by limiting the range in which overdetection is allowed to only environments where detection is high, the detection unit 13 can reduce overdetection while overlooking of equipment overall.

The detection criterion may be a threshold value for comparison with a certainty level indicating the likelihood that the equipment is included as a subject in the vehicle surroundings image. In this case, if the certainty level is equal to or greater than the criterion, the equipment is detected, and if the certainty level is lower than the criterion, the equipment is not detected. In other words, the lower the detection criterion, the higher the equipment detection rate, and although there is a possibility of an increase in overdetection, the number of missed detections is reduced. The memory unit 29 may store each environment that can be estimated (for example, inside a tunnel, not inside a tunnel) in association with the detection criterion.

The detection unit 13 uses lower detection standards when the vehicle is inside a tunnel than when the vehicle is not inside a tunnel. In other words, if the estimation result supplied from the estimation unit 12 indicates that the vehicle surroundings image was taken inside a tunnel, the detection unit 13 detects the equipment using lower standards. On the other hand, if the estimation result supplied from the estimation unit 12 indicates that the vehicle surroundings image was taken outside a tunnel, the detection unit 13 detects the equipment using higher standards.

As one example of a method for detecting equipment, the detection unit 13 receives an image of the vehicle's surroundings as input and detects the equipment using a region extraction model trained to output the area of the equipment detected by region extraction (e.g., PWC). As another example, the detection unit 13 receives an image of the vehicle's surroundings as input and detects the equipment using an object detection model trained to output the equipment detected by object detection (e.g., SSD (Single Shot Multibox Detector)). In this exemplary embodiment, an example in which an object detection model is applied as the detection model will be mainly described.

Furthermore, when detecting equipment using an object detection model, the detection unit 13 detects the equipment depending on whether the certainty level output from the object detection model exceeds a reference level. For example, when the estimation result supplied from the estimation unit 12 indicates that the vehicle surroundings image was taken inside a tunnel, the detection unit 13 detects the equipment depending on whether the certainty level output from the object detection model is higher than a first threshold. On the other hand, when the estimation result supplied from the estimation unit 12 indicates that the vehicle surroundings image was taken outside a tunnel, the detection unit 13 detects the equipment depending on whether the certainty level output from the object detection model is higher than a second threshold that is higher than the first threshold.

As an example, assume that the estimation result supplied from the estimation unit 12 indicates that the vehicle surroundings image was taken inside a tunnel, and the confidence level output from the object detection model is higher than the first threshold. In this case, the detection unit 13 detects the equipment detected by the object detection model as equipment included in the vehicle surroundings image.

On the other hand, assume that the estimation result supplied from the estimation unit 12 indicates that the vehicle surroundings image was taken inside a tunnel, and the confidence level output from the object detection model is equal to or less than the first threshold. In this case, the detection unit 13 does not detect the equipment detected by the object detection model as equipment included in the vehicle surroundings image.

As another example, consider a case where the estimation result supplied from the estimation unit 12 indicates that the vehicle surroundings image was taken outside a tunnel, and the confidence level output from the object detection model is lower than a second threshold that is higher than the first threshold. In this case, the detection unit 13 does not detect the equipment detected by the object detection model as equipment included in the vehicle surroundings image.

On the other hand, assume that the estimation result supplied from the estimation unit 12 indicates that the vehicle surroundings image was taken outside the tunnel, and the confidence level output from the object detection model is higher than the second threshold. In this case, the detection unit 13 detects the equipment detected by the object detection model as equipment included in the vehicle surroundings image.

(Image connection unit 21)
The image connection unit 21 connects images. For example, the image connection unit 21 connects a plurality of vehicle surroundings images stored in the storage unit 29. The image connection unit 21 stores the connected vehicle surroundings images in the storage unit 29.

For example, the image linking unit 21 acquires multiple vehicle surroundings images captured by camera CA1 from the storage unit 29. Next, the image linking unit 21 sorts the acquired multiple vehicle surroundings images according to the date and time of capture. The image linking unit 21 then links the sorted multiple vehicle surroundings images so that the equipment included as subjects in each of the multiple vehicle surroundings images is connected, thereby generating a linked vehicle surroundings image. The image linking unit 21 performs similar processing on the multiple vehicle surroundings images captured by the other cameras CA2 to CA6, respectively, to generate a linked vehicle surroundings image.

(Analysis Department 22)
The analysis unit 22 analyzes the state of the equipment. As an example, the analysis unit 22 analyzes the state of the equipment using a quality determination model that receives an equipment image as input and outputs information indicating whether the state of the equipment included as a subject in the equipment image is poor and the degree of certainty of the information.

As another example, in this exemplary embodiment, the analysis unit 22 uses parameters according to the environment estimated by the estimation unit 12 to analyze the state of the equipment by referring to an equipment image acquired from a vehicle surroundings image, the equipment image including the equipment included as a subject in the vehicle surroundings image.

In this configuration, the parameter may be a criterion for determining that equipment is defective. For example, the criterion for determining that equipment is defective may be a threshold value for comparison with a certainty level indicating the likelihood that the equipment is defective. In this case, if the certainty level is equal to or greater than the criterion, the equipment is determined to be defective, and if the certainty level is lower than the criterion, the equipment is determined to be not defective (i.e., good). In other words, the lower the criterion for determining that equipment is defective, the higher the defect determination rate, and although there is a possibility of an increase in erroneous determinations in which equipment that is actually good is determined to be defective, the number of times that equipment is overlooked as defective is reduced. The memory unit 29 may store each environment that can be estimated (for example, inside a tunnel, not inside a tunnel) in association with the criterion for determining that equipment is defective.

In this case, the analysis unit 22 determines whether the equipment is defective using a lower standard as the difficulty of determining whether it is defective increases depending on the environment at the time of photographing. For example, when analyzing equipment included as a subject in an equipment image in which the estimation unit 12 has estimated that the environment at the time of photographing was inside a tunnel, the analysis unit 22 uses the third threshold as the standard for determining that the equipment is defective. On the other hand, when analyzing equipment included as a subject in an equipment image in which the estimation unit 12 has estimated that the environment at the time of photographing was outside a tunnel, the analysis unit 22 uses a fourth threshold, which is higher than the third threshold, as the standard for determining that the equipment is defective.

As an example, we will explain a case where the analysis unit 22 uses the above-mentioned pass/fail judgment model. In this configuration, we will assume that an equipment image in which the estimation unit 12 estimates that the environment at the time of shooting was inside a tunnel is input to the pass/fail judgment model. In this case, if the confidence level output from the pass/fail judgment model is higher than the third threshold, the analysis unit 22 outputs the information output from the pass/fail judgment model indicating whether the condition of the equipment is poor or not as the analysis result. On the other hand, if the confidence level output from the pass/fail judgment model is lower than the third threshold, the analysis unit 22 outputs an analysis result indicating that pass/fail judgment is impossible.

Next, consider a case where an equipment image estimated by the estimation unit 12 to be taken in an environment other than a tunnel is input to the quality determination model. In this case, if the confidence level output from the quality determination model is higher than the fourth threshold, the analysis unit 22 outputs the information output from the quality determination model indicating whether the condition of the equipment is poor or not as the analysis result. On the other hand, if the confidence level output from the quality determination model is lower than the fourth threshold, the analysis unit 22 outputs an analysis result indicating that quality determination is impossible.

In this way, for equipment that appears as a subject in images of the vehicle's surroundings taken outside the tunnel, the analysis unit 22 sets higher standards for determining that the equipment is defective, thereby reducing over-detection, in which equipment that is not defective is determined to be defective. On the other hand, for equipment that appears as a subject in images of the vehicle's surroundings taken inside the tunnel, the analysis unit 22 sets lower standards for determining that the equipment is defective, thereby allowing over-detection while reducing the number of times that equipment defects are overlooked. Furthermore, by limiting the range in which over-detection is allowed to the inside of the tunnel, it is possible to suppress over-detection overall while reducing the number of times that defects are overlooked.

Furthermore, it is preferable that the third threshold and the fourth threshold differ depending on the equipment and its condition. With this configuration, the analysis unit 22 can appropriately analyze the condition for each piece of equipment and each condition of the equipment.

(Flow of information processing method S2)
The flow of the information processing method S2 according to this exemplary embodiment will be described with reference to Fig. 7. Fig. 7 is a flow chart showing the flow of the information processing method S2 according to this exemplary embodiment.

(Step S21)
In step S21, the acquisition unit 11 acquires the vehicle surroundings images captured by the cameras CA1 to CA6 and stores the acquired vehicle surroundings images in the storage unit 29.

(Step S22)
In step S22, the image connection unit 21 acquires a plurality of vehicle surroundings images stored in the storage unit 29. The image connection unit 21 also connects the acquired plurality of vehicle surroundings images. The image connection unit 21 stores the connected vehicle surroundings images in the storage unit 29.

(Step S23)
The estimation unit 12 acquires the post-coupling vehicle surroundings image stored in the storage unit 29. The estimation unit 12 estimates whether the environment at the time of image capture was inside a tunnel from the post-coupling vehicle surroundings image. The processing in step S23 will be described with reference to Fig. 8. Fig. 8 is a diagram showing an example of the processing by the estimation unit 12 according to this exemplary embodiment.

As shown in FIG. 8, the estimation unit 12 acquires the concatenated vehicle surroundings image c_vp. Next, as shown in the upper part of FIG. 8, the estimation unit 12 divides the concatenated vehicle surroundings image c_vp into multiple split images (split image dp1, split image dp2) along a direction corresponding to the traveling direction of the railway vehicle. In the upper part of FIG. 8, the estimation unit 12 similarly divides the right side of split image dp2 into multiple split images. Furthermore, as shown in the upper part of FIG. 8, the estimation unit 12 may divide the concatenated vehicle surroundings image c_vp into multiple split images so that at least some areas overlap.

Next, the estimation unit 12 cuts out a partial image from the divided image. As an example, the estimation unit 12 is configured to cut out a partial image that includes at least one piece of equipment as a subject, based on the angles at which cameras CA1 to CA6 take pictures, the position of the equipment, etc. For example, as shown in Figure 8, a partial image tp1 that includes a trolley wire as a subject is cut out from divided image dp1.

The estimation unit 12 then estimates, for each partial image, whether the environment at the time of shooting was inside a tunnel. The method by which the estimation unit estimates whether the environment at the time of shooting was inside a tunnel is as described above. The estimation unit 12 supplies the partial image and the estimation results to the detection unit 13.

Here, the estimation unit 12 may correct the environment at the time of shooting for each partial image based on the order of the environments at the time of shooting estimated for each partial image. For example, the estimation unit 12 assumes that a sequence of N or more partial images estimated to be a first environment, a sequence of n or less partial images estimated to be a second environment, and a sequence of N or more partial images estimated to be the first environment are arranged in this order. n is an integer equal to or greater than 1, and N is an integer greater than n. The first environment is, for example, "inside a tunnel," and the second environment is, for example, "outside a tunnel."

In this case, the estimation unit 12 may correct the estimated environment at the time of shooting for each of the n or fewer partial images that were estimated to be the second environment to the first environment. For example, in Figure 8, consider a case where the environment at the time of shooting of one partial image tp10 is estimated to be outside the tunnel, and the environments at the time of shooting of the two adjacent partial images (partial images tp8-tp9, tp11-tp12) are estimated to be inside the tunnel. In this case, since the estimation result that only partial image tp10 was outside the tunnel may be incorrect, the estimation unit 12 may correct the environment at the time of shooting of partial image tp10 and estimate it to be inside the tunnel.

In this way, if it is estimated that the environment when a certain partial image (partial image tp11 in Figure 8) was photographed was different from the environment when the adjacent partial images (partial image tp10 and partial image tp12) were photographed, it is unlikely that the environment when only the certain partial image was photographed was different, and therefore it is highly likely that the estimation result for that certain partial image will be incorrect. In such a case, the estimation unit 12 corrects the environment when the certain partial image was photographed so that it is the same as the environment when the adjacent partial images were photographed. Therefore, the estimation unit 12 can improve the accuracy of estimating the environment when the partial image was photographed.

Furthermore, consider a case where the captured image of the vehicle's surroundings spans multiple environments. For example, partial image tp8 shown in Figure 8 is partly an image outside the tunnel and partly an image inside the tunnel. In this case, instead of the above-mentioned estimation model, the estimation unit 12 may use an estimation model that, when a partial image is input, outputs information indicating the area inside the tunnel in the partial image. The estimation unit 12 references the information output from the estimation model, and if the area inside the tunnel is equal to or greater than a predetermined value, estimates that the partial image was captured inside the tunnel. With this configuration, the estimation unit 12 can preferably detect facilities inside the tunnel in the facility detection process described below.

As another example, the estimation unit 12 may be configured to reference the confidence level output from the estimation model described above. For example, when a partial image is input, if the confidence level for estimating that the image is inside a tunnel based on the estimation model is higher than the confidence level for estimating that the image is not inside a tunnel, the estimation unit 12 estimates that the partial image was taken inside the tunnel. On the other hand, if the confidence level for estimating that the image is inside a tunnel based on the estimation model is lower than the confidence level for estimating that the image is not inside a tunnel, the estimation unit 12 estimates that the partial image was taken outside the tunnel. Even with this configuration, the estimation unit 12 can effectively detect equipment inside a tunnel in the equipment detection process described below.

As yet another example, the estimation unit 12 may estimate whether a partial image is an image taken inside a tunnel based on pixel values. For example, the estimation unit 12 calculates the percentage of pixels in the partial image that have pixel values lower than a predetermined value. If the calculated percentage is equal to or greater than a predetermined percentage (e.g., 30% or greater), the estimation unit 12 may estimate that the environment in which the partial image was taken is inside a tunnel. Even with this configuration, the estimation unit 12 can preferably detect facilities inside a tunnel in the facility detection process described below.

(Step S24)
In step S24, the detection unit 13 refers to the estimation result supplied from the estimation unit 12 in step S23 and determines whether the environment in which the partial image supplied from the estimation unit 12 was photographed was estimated to be inside a tunnel.

(Step S25)
If it is estimated in step S24 that the subject is inside a tunnel (step S24: YES), in step S25 the detection unit 13 detects equipment included as a subject from the partial image using a low criterion. The detection unit 13 supplies the equipment image including the detected equipment and the estimation result supplied from the estimation unit 12 to the analysis unit 22.

(Step S26)
If it is estimated in step S24 that the subject is not inside a tunnel (step S25: NO), the detection unit 13 detects equipment included as a subject from the partial image using a high standard in step S26. The detection unit 13 supplies the equipment image including the detected equipment and the estimation result supplied from the estimation unit 12 to the analysis unit 22.

(Step S27)
In step S27, the analysis unit 22 analyzes the state of the equipment by referring to the equipment image, using the parameters according to the environment estimated by the estimation unit 12.

(Effects of information processing device 2)
In this way, the information processing device 2 according to this exemplary embodiment estimates whether the environment at the time the vehicle surroundings image was captured is inside a tunnel, and detects equipment using lower criteria as the difficulty of detection increases depending on whether the environment is inside a tunnel or not.

In images of the vehicle's surroundings taken inside a tunnel, equipment may be difficult to see due to darkness, or the pattern of the tunnel's interior wall in the background of the equipment may resemble the equipment, making it difficult to detect the equipment from the vehicle's surroundings image. The information processing device 2 detects equipment included as subjects in images of the vehicle's surroundings taken inside a tunnel using low standards, thereby effectively preventing equipment included as subjects in images of the vehicle's surroundings taken inside a tunnel from being overlooked.

On the other hand, for images of the vehicle's surroundings taken outside the tunnel, the information processing device 2 detects equipment using high standards. As a result, the information processing device 2 can effectively prevent overdetection of subjects other than equipment as equipment.

[Software implementation example]
Some or all of the functions of the information processing devices 1 and 2 may be realized by hardware such as an integrated circuit (IC chip), or by software.

In the latter case, information processing devices 1 and 2 are realized, for example, by a computer that executes program instructions, which are software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 9. Computer C has at least one processor C1 and at least one memory C2. Memory C2 stores program P for operating computer C as information processing devices 1 and 2. In computer C, processor C1 reads and executes program P from memory C2, thereby realizing each function of information processing devices 1 and 2.

The processor C1 may be, for example, a CPU (Central Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating point number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination of these. The memory C2 may be, for example, a flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination of these.

Computer C may further include RAM (Random Access Memory) for expanding program P during execution and for temporarily storing various data. Computer C may also include a communications interface for sending and receiving data to and from other devices. Computer C may also include an input/output interface for connecting input/output devices such as a keyboard, mouse, display, or printer.

Furthermore, the program P can be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, disk, card, semiconductor memory, or programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communications network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[Additional Note 1]
The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiments are also included in the technical scope of the present invention.

[Additional Note 2]
Some or all of the above-described embodiments can also be described as follows: However, the present invention is not limited to the following described aspects.

(Appendix 1)
An information processing device including: an acquisition means for acquiring a vehicle surroundings image; an estimation means for estimating the environment at the time of photographing by referring to the vehicle surroundings image; and a detection means for detecting equipment included as a subject from the vehicle surroundings image using a lower criterion as the difficulty of detection according to the estimated environment increases.

(Appendix 2)
The information processing device described in Appendix 1 further includes an analysis means for analyzing the state of the equipment by referring to an equipment image obtained from the vehicle surroundings image, the equipment image including the equipment included as a subject in the vehicle surroundings image, using parameters corresponding to the estimated environment.

(Appendix 3)
The information processing device described in Appendix 1 or 2, wherein the estimation means estimates whether the environment at the time of shooting is inside a tunnel or not, and the detection means uses a lower standard for the detection when the environment is inside a tunnel than when the environment is not inside a tunnel.

(Appendix 4)
The information processing device described in any of Appendices 1 to 3, wherein the estimation means divides the vehicle surroundings image into a plurality of partial images along a direction corresponding to the vehicle's traveling direction, and corrects the environment at the time of shooting for each partial image based on the order of the environment at the time of shooting estimated for each partial image.

(Appendix 5)
An information processing method including an information processing device acquiring a vehicle surroundings image, estimating the environment at the time of photographing by referring to the vehicle surroundings image, and detecting equipment included as a subject from the vehicle surroundings image using lower criteria as the difficulty of detection according to the estimated environment increases.

(Appendix 6)
The information processing method described in Appendix 5 further includes the information processing device analyzing the state of the equipment by referring to an equipment image obtained from the vehicle surroundings image, the equipment image including the equipment included as a subject in the vehicle surroundings image, using parameters corresponding to the estimated environment.

(Appendix 7)
A program that causes a computer to function as an information processing device, the program causing the computer to function as an acquisition means that acquires an image of the vehicle's surroundings, an estimation means that estimates the environment at the time of shooting by referring to the image of the vehicle's surroundings, and a detection means that detects equipment included as a subject from the image of the vehicle's surroundings using lower criteria as the difficulty of detection increases according to the estimated environment.

(Appendix 8)
The program described in Appendix 7 further causes the computer to function as an analysis means for analyzing the state of equipment by referring to an equipment image obtained from the vehicle surroundings image, the equipment image including equipment that is included as a subject in the vehicle surroundings image, using parameters according to the estimated environment.

[Appendix 3]
Some or all of the above-described embodiments can also be expressed as follows.

An information processing device comprising at least one processor, which executes an acquisition process for acquiring a vehicle surroundings image, an estimation process for estimating the environment at the time of image capture by referencing the vehicle surroundings image, and a detection process for detecting equipment included as a subject in the vehicle surroundings image using lower criteria as the difficulty of detection increases according to the estimated environment.

The information processing device may further include a memory, and this memory may store a program for causing the processor to execute the acquisition process, the estimation process, and the detection process. The program may also be recorded on a computer-readable, non-transitory, tangible recording medium.

1, 2 Information processing device 11 Acquisition unit 12 Estimation unit 13 Detection unit 22 Analysis unit

Claims (8)

an acquisition means for acquiring an image of the surroundings of the vehicle;
an estimation means for estimating an environment at the time of capturing the vehicle surroundings image by referring to the vehicle surroundings image;
a detection means for detecting equipment included as a subject in the vehicle surroundings image using a lower criterion as the difficulty of detection according to the estimated environment increases; and
An information processing device comprising: and further comprising an analysis means for analyzing a state of an equipment by referring to an equipment image acquired from the vehicle surroundings image, the equipment image including the equipment included as a subject in the vehicle surroundings image, using parameters according to the estimated environment.
The information processing device according to claim 1 . the estimation means estimates whether the environment at the time of photographing is inside a tunnel,
the detection means uses a lower standard for the detection when the vehicle is inside a tunnel than when the vehicle is not inside a tunnel;
3. The information processing device according to claim 1. The estimation means
Dividing the vehicle surroundings image into a plurality of partial images along a direction corresponding to a traveling direction of the vehicle, and correcting the environment at the time of photographing each partial image based on the order of the environment at the time of photographing estimated for each partial image.
3. The information processing device according to claim 1. The information processing device
Acquiring an image of the vehicle's surroundings;
estimating an environment at the time of capturing an image by referring to the vehicle surroundings image;
detecting the facility using a lower criterion as a criterion for detecting the facility included as a subject from the vehicle surroundings image, the higher the difficulty of detection according to the estimated environment; and
An information processing method, including: The information processing device includes:
and analyzing a state of the equipment by referring to an equipment image acquired from the vehicle surroundings image, the equipment image including the equipment included as a subject in the vehicle surroundings image, using parameters according to the estimated environment.
The information processing method according to claim 5 . A program that causes a computer to function as an information processing device,
The computer
an acquisition means for acquiring an image of the surroundings of the vehicle;
an estimation means for estimating an environment at the time of capturing the vehicle surroundings image by referring to the vehicle surroundings image;
a detection means for detecting equipment included as a subject in the vehicle surroundings image using a lower criterion as the difficulty of detection according to the estimated environment increases; and
A program that functions as a The computer
and further functioning as an analysis means for analyzing a state of equipment by referring to an equipment image acquired from the vehicle surroundings image, the equipment image including the equipment included as a subject in the vehicle surroundings image, using the parameters according to the estimated environment.
The program according to claim 7.