WO2025173166A1 - Apparatus, operability estimating method, and program - Google Patents

Abstract

This apparatus makes it possible to estimate operability with regard to remote operation of a work machine, taking into account communication delay and video quality, by including: a delay operability estimating unit configured to calculate a delay operability, which is an estimated value of operability based on a delay, on the basis of the characteristic that the operability of the work machine decreases as a delay in communication relating to operation during remote operation of the work machine increases, but the operability does not decrease if the delay is less than a predetermined time; a video operability estimating unit configured to calculate a video operability, which is an estimated value of operability based on the quality of a video, on the basis of the characteristic that, in relation to communication of a video of the work machine during remote operation, operability is high until a certain bit rate, and operability decreases as the packet loss rate increases; and an integrating unit configured to calculate an estimated value of the operability of the work machine on the basis of the delay operability and the video operability.

Description

Apparatus, operability estimation method, and program

The present invention relates to a device, an operability estimation method, and a program.

In order to reduce the risk of accidents during on-site work and alleviate labor shortages, remote control systems are being offered that allow heavy machinery and other construction equipment in remote locations to be operated from a control room. With these remote control systems, it is necessary to consider the risk of accidents due to reduced operability caused by a decline in network quality during operation.

This section explains the impact of a decline in network quality during remote operation on operability. During remote operation, operation information from an operating device in the control room is sent to a remote work machine, causing the work machine to operate, and the operation of the work machine is then transmitted to the control room as a camera image. Therefore, a decline in the quality (communication quality) of the network that transmits the operation information and images can potentially reduce operability for the operator. For example, an increase in packet loss due to a decline in network quality, or a decline in image quality due to a decrease in encoding bit rate associated with a decrease in throughput, can make it difficult to visually grasp the situation in a remote location, reducing operability. Furthermore, increased network latency increases the reaction time of the work machine in the image to the operator's operation, creating a discrepancy between the operator's operation and the work machine's operation in the image, reducing operability (Non-Patent Document 1).

Given the above impacts, it is important to quickly grasp changes in video quality and latency, as well as any degradation in operability caused by a combination of these. Therefore, technology is needed to obtain information on video quality and latency that cause degradation in operability from information on the devices and networks that make up the remote control, and estimate operability by understanding the impact of this information on operability. A similar method for estimating operability has been established by ITU-T in Recommendation G.1072 (Non-Patent Document 2), which targets cloud gaming, a service operated via a network, and estimates the quality experienced by users during use from latency, bit rate, packet loss rate, etc.

"Joint Research Report on Improving Remote Control Technology for Construction Machinery," Public Works Research Institute, 2016 G.1072: Opinion model predicting gaming quality of experience for cloud gaming services, 2020, ITU-T

When remotely operating work machinery such as heavy machinery, it takes a certain amount of time for the operation to be reflected in the operation of the work machine due to factors such as the resistance of the control lever, the weight of the work machine, and hydraulic pressure. For this reason, delay has the characteristic of having almost no effect on operability as long as it is below a certain delay time (Non-Patent Document 1). However, while Non-Patent Document 1 indicates the effect that changes in delay have on operability, it does not consider the overall effect on operability due to changes in delay and video quality. Therefore, Non-Patent Document 1 cannot make estimates that take into account the effect that changes in video quality have on operability.

While Non-Patent Document 2 takes into account the overall impact of delay and video quality on operability, it fails to consider the impact that changes in delay have on operability. As a result, when attempting to estimate operability degradation due to delay based on Non-Patent Document 2, operability will be underestimated when delay is low. In Figure 1, the solid curve is a curve that shows actual operability relative to delay. The dashed curve is a curve that shows the relationship between delay and operability estimated based on Non-Patent Document 2. "Detectable delay" is the minimum amount of delay at which a difference in operability from when the delay is zero can be detected. Figure 1 shows that Non-Patent Document 2 estimates operability to be smaller than the actual operability for delays less than the "detectable delay" that do not actually result in any degradation in operability.

Furthermore, the overall impact of latency and video quality on operability is such that, when the latency is small, degradation of video quality has a large impact on operability, whereas when the latency is large, degradation of video quality has a smaller impact (Figure 2). However, in Non-Patent Document 2, the impact of degradation of video quality on operability is constant regardless of the magnitude of the latency. Therefore, the overall impact of latency and video quality on operability is not taken into consideration.

The present invention was made in consideration of the above points, and aims to make it possible to estimate the operability of remotely operating a work machine while taking into account communication delays and video quality.

In order to solve the above problem, the device has a delay operability estimation unit configured to calculate delay operability, which is an estimate of operability based on the delay, based on the characteristic that the greater the delay in communication related to operation during remote operation of a work machine, the lower the operability of the work machine, but that the operability does not decrease if the delay is less than a predetermined time; a video operability estimation unit configured to calculate video operability, which is an estimate of operability based on the quality of the video, based on the characteristic that, with regard to communication of video of the work machine during remote operation, operability increases up to a certain bit rate, and the higher the packet loss rate, the lower the operability; and an integration unit configured to calculate an estimate of the operability of the work machine based on the delay operability and the video operability.

It is possible to estimate the operability of remotely operating a work machine, taking into account communication delays and video quality.

FIG. 10 is a diagram for explaining the relationship between delay and operability. FIG. 10 is a diagram for explaining the influence of delay and video quality on operability. 1 is a diagram illustrating an example of a configuration of a remote control system 20 according to an embodiment of the present invention. 1 is a diagram illustrating an example of a hardware configuration of an operability estimation device 10 according to an embodiment of the present invention. 1 is a diagram illustrating an example of a functional configuration of an operability estimation device 10 according to an embodiment of the present invention. 10 is a flowchart illustrating an example of a processing procedure executed by the operability estimation device 10.

Embodiments of the present invention will now be described with reference to the drawings. Figure 3 is a diagram showing an example configuration of a remote control system 20 in an embodiment of the present invention. In Figure 3, the remote control system 20 is a computer system that enables a work machine 221, such as heavy machinery, located in a remote location 22 to operate from a remote control room 21. An operation device 211 and a monitoring terminal 212 are located in the remote control room 21. A work machine 221 and an imaging device 222 are located in the remote location 22.

The operation device 211 is a device that allows the worker to remotely operate the work machine 221. When the worker operates the operation device 211, the operation information is transmitted via the network 23, and the work machine 221 receives the operation information. The work machine 221 operates in accordance with the operation information. In order to check the operation of the work machine 221 and the status of the work site from the remote control room 21, the imaging device 222 installed on the work machine 221 acquires (captures) video of the work machine 221. The captured video is encoded, and the encoded video (hereinafter referred to as "encoded video") is transmitted via the network 23 to the monitoring terminal 212. The monitoring terminal 212 displays video that has been decoded from the encoded video. The worker can check the work status by viewing the video.

The operability estimation device 10 is one or more computers that output an estimated value of the operability of remotely operating the work machine 221 (hereinafter referred to as "operability o") by inputting information related to video quality obtainable from the remote operation system 20 and information related to communication delays via the network 23. The operability estimation device 10 may be located outside the remote operation system 20, or may be located inside the remote operation system 20, such as in the remote operation room 21.

FIG. 4 is a diagram showing an example of the hardware configuration of an operability estimation device 10 according to an embodiment of the present invention. The operability estimation device 10 in FIG. 4 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a processor 104, and an interface device 105, all of which are interconnected via a bus B.

The program that realizes the processing in the operability estimation device 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is inserted into the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101; it may be downloaded from another computer via the network 23. The auxiliary storage device 102 stores the installed program as well as necessary files, data, etc.

When an instruction to start a program is received, the memory device 103 reads the program from the auxiliary storage device 102 and stores it. The processor 104 is a CPU or a GPU (Graphics Processing Unit), or a CPU and a GPU, and executes functions related to the operability estimation device 10 in accordance with the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to the network 23.

FIG. 5 is a diagram showing an example of the functional configuration of an operability estimation device 10 according to an embodiment of the present invention. In FIG. 5, the operability estimation device 10 has an input unit 11, a delay operability estimation unit 12, a video operability estimation unit 13, an integration unit 14, and an output unit 15. Each of these units is realized by the processor 104 executing one or more programs installed in the operability estimation device 10.

The processing procedures executed by the operability estimation device 10 are described below. Figure 6 is a flowchart illustrating an example of the processing procedures executed by the operability estimation device 10.

In step S101, the input unit 11 inputs information about video quality and delay from the remote control system 20.

In this embodiment, the bit rate of the encoded video and the packet loss rate during transmission of the encoded video are examples of information about video quality, and the delay time, which is the time from when an operation is input to the operation device 211 until the operation of the work machine 221 can be visually confirmed on the video displayed by the monitoring terminal 212 (strictly speaking, "the time until the work machine 221 starts operating in response to the input of an operation" + "the time until the video of the work machine 221 starting to operate is reflected on the monitor in the remote control room 21") is described as an example of information about delay. These bit rate, packet loss rate, and delay time are input to the operability estimation device 10.

)。 The delay time may be measured as the difference between the time when the operation of the operation device 211 begins and the time when the image changes accordingly (for example, the difference between the time when the lever operation for the turning operation begins and the time when the image begins to change due to the turning). Alternatively, the delay time may be the sum of the measurement result of the round trip time of a test packet such as a Ping on the network 23 (because there are two routes, from the operation device 211 to the work machine 221 and from the imaging device 222 to the monitoring terminal 212, for example, the average RTT of these two routes) and the buffer time of each receiving terminal for the encoded image or operation information, and the processing time of the image encoding device and decoding device. In either case, the delay time is input as the average measured time in units of one second. Methods other than these may also be used to input the delay time. The receiving terminal for the encoded image is the decoding device. The decoding device may be built into the monitoring terminal 212 or may be connected externally to the monitoring terminal 212. The receiving terminal for the operation information is the work machine 221. Similarly, the video encoding device may be built into the imaging device 222 or may be externally connected to the imaging device 222.

The bit rate of the encoded video is determined using the amount of data transmitted in the encoded video packets and setting information obtained from the encoding device and decoding device in the remote control system 20. The amount of data transmitted can be calculated, for example, from the payload size of the video packet per unit time. Setting information is, for example, the bit rate targeted by the encoding device when encoding. In this case, this bit rate may be used as the bit rate of the encoded video.

The packet loss rate is calculated based on the ratio of the amount of transmitted data to the amount of received data for encoded video packets, or the packet loss rate per second calculated based on the results of sending and receiving test packets such as Ping on the network 23 that transfers encoded video within the remote control system 20. Note that other bit rates and packet loss rates may also be used.

Next, the delay operability estimation unit 12 executes a process in steps S102 and S103 for calculating a delay operability o delay , which is an estimated value of delay-based operability, based on the delay _time at unit time intervals (the average measurement time in 1-second units is the delay time) d [ms]. Below, a method for calculating the delay operability o _delay from the delay time d [ms] will be described.

Operability is negatively correlated with delay, with operability decreasing the greater the delay. On the other hand, there is a characteristic (hereinafter referred to as "characteristic a") whereby if the delay is less than a predetermined time (hereinafter referred to as "detection limit t"), the operator cannot detect the delay and operability does not decrease. The delay operability estimation unit 12 takes this characteristic a into consideration.

First, in step S102, the delay operability estimation unit 12 selects the value of the detection limit t, which serves as a threshold. The value of t may be preset as a constant based on characteristics previously investigated through subjective evaluation experiments, or may be adjusted by the delay operability estimation unit 12 using an arbitrary coefficient to take into account the influence of video quality, or may be variable.

Next, the delay operability estimation unit 12 calculates the delay operability o _delay by applying the delay time d [ms] and the detection limit t to one of the following equations (1-1) to (1-3), which is a model in which if the delay time is less than t, the value of the delay operability o _delay is the same as in a state without delay, and if the delay time exceeds the detection limit t, the delay operability o _delay decreases (S103).

In each equation, c ₁ and c ₂ are coefficients that differ for each equation. Note that other equations may be used that take into account the characteristic a.

The delay operability o _delay is a value obtained by normalizing the delay-based operability to the range of 0-1.

Next, the video operability estimation unit 13 calculates the video operability o video, which is an estimated value of operability based on the video quality (encoded video bit rate, packet loss rate), based on the bit rate b [kbps] of the encoded _video and the packet loss rate p [%] (S104).

Video quality correlates with bit rate, and the higher the bit rate, the higher the operability. On the other hand, when the bit rate reaches a certain level, the video quality almost reaches its upper limit, and operability does not change even at bit rates above that level. In other words, operability increases up to a certain bit rate. Therefore, using a model in which operability increases as the bit rate increases but does not change above a certain bit rate, video operability (o _{video )} is calculated when p = 0. In addition, with regard to the impact of packet loss, the packet loss rate is negatively correlated with operability, and the higher the loss rate, the lower the operability. Furthermore, in cases where the number of packet losses is more negatively correlated with operability than the packet loss rate, the higher the bit rate for the same packet loss rate, the greater the decrease in operability.

Therefore, the video operability estimation unit 13 takes into consideration the above characteristic (hereinafter referred to as "characteristic b") and calculates the video operability o video based on the magnitude of the bit rate when the packet loss rate is 0, and calculates the video operability o _video from the bit rate and packet loss rate when the packet loss rate is greater than 0, based on one of the following equations (2-1) to (2-5 ₎ .

Here, in each equation, c3 _{to c8} are coefficients that differ for each equation. Also, o _max is the maximum value that video operability o _video can take. That is, the value range of video operability o _video is 1 to 5. The value of o _max may be the maximum value of operability o obtained in a range of 1 to 5 in a prior subjective evaluation experiment, or it may be 5, which is the theoretical maximum value of operability o.

Note that other equations that take characteristic b into account may also be used.

Next, the integration unit 14 calculates the operability o of the entire remote operation system 20 based on both the delay and the video quality, based on the delay operability o _delay and the video operability o _video (S105).

The operability o has a characteristic (hereinafter referred to as "characteristic c") of decreasing synergistically with respect to the delay operability o _delay due to the influence of a decrease in video quality, as shown in Figure 2. Here, "synergistic" does not mean that the operability o decreases while maintaining the difference in video quality relative to the overall delay, but rather expresses a tendency that when the delay decreases due to the influence of video quality, in the case of a relatively large delay, the operability o of the delay does not decrease even if the video quality decreases, but in the case of a relatively small delay, the operability o of the delay decreases when the video quality decreases. In other words, when the delay is small, the difference in video quality is reflected in the operability o, but once the operability o relative to the delay begins to decrease, the slope of the decrease in operability o changes depending on the video quality. This tendency is expressed as "synergistic."

The integration unit 14 calculates the operability o based on the following formula (3-1) or formula (3-2), taking into consideration the synergistic influence of the decrease in video operability o _video and the decrease in delay operability o _delay .

Here, in each equation, c ₉ is a coefficient that differs for each equation. o _max is as described above. The value range of operability o is 1-5, with a larger value indicating higher operability. Each equation includes a term in which video operability o _video and delay operability o _delay are multiplied. This allows the above synergistic effect to be reflected in each equation. Note that other equations that take characteristic c into account may also be used.

The output unit 15 then performs output based on the operability o (S106). For example, the output unit 15 may output (display) the operability o to a specified terminal. Alternatively, the output unit 15 may output control information for the remote control system 20. For example, if the operability o is equal to or less than a certain value, the output unit 15 may transmit control information to the remote control system 20 to stop the remote control system 20.

The values of t and each coefficient are derived by optimizing the input and the results of a subjective evaluation of operability for that input using a method such as the least squares method. The values of t and each coefficient may be derived using a method other than the above derivation method, or predetermined coefficients may be used.

As described above, this embodiment makes it possible to estimate the operability o for the remote operation of the work machine 221, taking into account communication delays and video quality. By monitoring the operability o, if the operability o continues to drop to a certain value, it becomes possible to notify the operator in the remote control room 21 to stop operation, to make an emergency stop on the work machine 221, or to notify the remote operation manager, thereby contributing to the prevention of accidents and other issues caused by a decline in the operator's operability.

Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention as set forth in the claims.

10 Operability estimation device 11 Input unit 12 Delay operability estimation unit 13 Video operability estimation unit 14 Integration unit 15 Output unit 20 Remote operation system 21 Remote operation room 22 Remote location 100 Drive device 101 Recording medium 102 Auxiliary storage device 103 Memory device 104 Processor 105 Interface device 211 Operation device 212 Monitoring terminal 221 Work machine 222 Imaging device B Bus

Claims (6)

a delay operability estimation unit configured to calculate a delay operability, which is an estimated value of operability based on a characteristic that the greater the delay in communication related to operation in remote operation of a work machine, the more the operability of the work machine deteriorates, but that the operability does not deteriorate if the delay is less than a predetermined time;
a video operability estimation unit configured to calculate video operability, which is an estimated value of operability based on the quality of the video, based on a characteristic that operability is high up to a certain bit rate and the operability decreases as the packet loss rate increases, with respect to communication of the video of the work machine during the remote operation;
an integrating unit configured to calculate an estimate of operability of the work machine based on the delay operability and the video operability;
An apparatus comprising: The video operability estimation unit is further configured to calculate the video operability based on a characteristic that the higher the bit rate for the same packet loss rate, the greater the degradation of operability.
2. The device of claim 1. the integrating unit is configured to calculate an estimated value of the operability of the work machine based on a characteristic that, when the operability of the work machine in response to the delay begins to deteriorate, the slope of the deterioration in the operability varies depending on the quality of the video.
2. The device of claim 1. calculating an estimated value of the operability of the work machine based on a communication delay time related to operation during remote operation of the work machine, and video bit rate information and a video packet loss rate related to video of the work machine during the remote operation;
An apparatus characterized in that a delay operability estimation procedure for calculating a delay operability, which is an estimated value of operability based on a delay, based on the characteristic that the greater the delay in communication related to operation in remote operation of a work machine, the more the operability of the work machine deteriorates, but if the delay is less than a predetermined time, the operability does not deteriorate;
a video operability estimation step of calculating video operability, which is an estimated value of operability based on the quality of the video, based on the characteristic that operability is high up to a certain bit rate with respect to communication of the video of the work machine during the remote operation, and the higher the packet loss rate, the lower the operability;
an integration step of calculating an estimate of the operability of the work machine based on the delay operability and the video operability;
The operability estimation method is characterized in that the above steps are executed by a computer. a delay operability estimation procedure for calculating a delay operability, which is an estimated value of operability based on a delay, based on the characteristic that the greater the delay in communication related to operation in remote operation of a work machine, the more the operability of the work machine deteriorates, but if the delay is less than a predetermined time, the operability does not deteriorate;
a video operability estimation step of calculating video operability, which is an estimated value of operability based on the quality of the video, based on the characteristic that operability is high up to a certain bit rate with respect to communication of the video of the work machine during the remote operation, and the higher the packet loss rate, the lower the operability;
an integration step of calculating an estimate of the operability of the work machine based on the delay operability and the video operability;
A program characterized by causing a computer to execute the above.