WO2024189997A1 - Information processing device - Google Patents

Abstract

An information processing device (10) comprises: an acquisition unit (11) that acquires information to be input into an action value function for estimating the value of an action related to a rearrangement of vehicles between a plurality of ports in a shared transportation service whereby the vehicles are shared; and a learning inference unit (12) that implements reinforcement learning with regard to the action value function using the information acquired by the acquisition unit (11) in order to obtain an action value function for selecting an action that will have the highest future action value.

Description

Information processing device

The present disclosure relates to an information processing device that optimizes the reallocation of vehicles among multiple ports in a shared transportation service. Note that a "port" refers to a space for parking multiple shared vehicles (such as a bicycle parking lot for parking multiple shared electric bicycles), and multiple ports are provided in multiple locations to provide a shared transportation service.

There is a known technology for vehicle sharing services that improves the vehicle utilization rate throughout the entire service by collecting vehicles from a port with a surplus of vehicles among multiple ports and placing the collected vehicles in a port with a shortage of vehicles (see Patent Document 1).

JP 2022-175898 A

The vehicle sharing services mentioned above are provided by private companies, so it is desirable to operate them in a way that maximizes the rewards (profits) of the service as a whole, rather than simply allowing ports to replenish bicycle shortages appropriately.

However, no technology has been proposed that reallocates vehicles with a primary focus on maximizing the overall reward (profit) of the sharing service.

The present disclosure takes into consideration the above circumstances and aims to optimize vehicle reallocation between multiple ports in a shared transportation service so as to maximize rewards (profits).

One type of machine learning known as reinforcement learning is the problem of an agent in an environment observing the current state and determining the future actions to be taken based on a policy from the state information obtained from the observation. The agent obtains rewards from the environment by executing the determined actions, and reinforcement learning learns the policy that will obtain the greatest rewards through a series of actions.

The applicant has focused on the above-mentioned reinforcement learning and has devised an invention that utilizes a reinforcement learning framework to optimize vehicle reallocation between multiple ports in a shared transportation service so as to maximize the reward in reinforcement learning, and discloses this invention in the present specification.

The information processing device according to the present disclosure includes an acquisition unit that acquires information to be input into an action value function that estimates the value of an action related to the reallocation of vehicles between multiple ports in a shared transportation service in which vehicles are shared, and a learning and inference unit that performs reinforcement learning on the action value function using the information acquired by the acquisition unit in order to make the action value function select an action that will maximize the value of a future action.

In the above information processing device, the acquisition unit acquires information to be input into an action value function that estimates the value of an action related to the reallocation of vehicles between multiple ports in a shared transportation service. Examples of the information will be described in detail in the embodiments of the invention, but "acquisition" here broadly includes not only obtaining information but also calculating information using a predetermined formula or the like. The learning and inference unit performs reinforcement learning on the action value function using the information acquired by the acquisition unit to turn the above action value function into an action value function that selects an action that will have the highest value for a future action. As a result, through reinforcement learning, the action value function learns to become an action value function that selects an action that will have the highest value for a future action.

The value of actions related to vehicle reallocation between multiple ports is estimated based on such an action value function, and the action that will have the highest value in the future is selected. Therefore, when reallocating vehicles between multiple ports in a shared transportation service, vehicle reallocation can be optimized to maximize the reward in reinforcement learning.

According to the present disclosure, when reallocating vehicles between multiple ports in a shared transportation service, it is possible to optimize the reallocation of vehicles so as to maximize rewards in reinforcement learning.

1 is a functional block diagram of an information processing device according to an embodiment of the present invention; FIG. 1 is a diagram for explaining an overview of reinforcement learning. FIG. 11 is a diagram showing an example of state information etc. when the present invention is applied to reinforcement learning. FIG. 13 is a diagram for explaining a discounted cumulative reward. FIG. 13 is a diagram for explaining a state value function. FIG. 13 is a diagram for explaining learning of a state value function. FIG. 11 is a diagram for explaining an example of calculating a reward in reinforcement learning. 13 is a diagram for explaining calculation of the average number of times used by placing it on a port. FIG. 13 is a diagram showing an example of information used in calculating the average number of times a device is used by being placed on a port. FIG. FIG. 13 is a diagram for explaining calculation of the average number of impressions of a bicycle. FIG. 13 is a diagram showing an example of information used in calculating the average number of impressions of a bicycle. FIG. 13 is a flow diagram showing the processing in the learning stage. FIG. 13 is a flowchart showing details of the process in step S4 of FIG. 12. FIG. 1 is a flow diagram showing the processing of the inference stage. FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device.

Below, we will explain one embodiment of an information processing device that optimizes vehicle reallocation between multiple ports in a shared transportation service, with reference to the drawings.

(Configuration of information processing device)
As shown in Fig. 1, the information processing device 10 includes an acquisition unit 11 and a learning and inference unit 12. The acquisition unit 11 is a functional unit having a function of acquiring information to be input to an action value function that estimates the value of an action related to the reallocation of vehicles among multiple ports in a shared transportation service in which vehicles are shared. The learning and inference unit 12 is a functional unit having a function of performing reinforcement learning on the action value function using the information acquired by the acquisition unit 11 in order to make the action value function select an action that will maximize the value of a future action.

In the shared transportation service of this embodiment, the vehicles shared are vehicles that can be restarted by replacing the battery and have advertisements placed on the body, and examples of such vehicles include rental electric bicycles, rental electric kick scooters, rental motorbikes, and rental cars. In the following, we will use as an example a service for sharing "rental electric bicycles," which will be abbreviated to "bicycles."

Furthermore, to provide a shared transportation service, multiple ports (here, bicycle parking areas for multiple shared bicycles) are provided in multiple locations, and multiple bicycles are parked at each port. Service staff use bicycle transport trucks (hereafter referred to as "relocation trucks") to "dispatch" bicycles to a port, "collect" bicycles from a port, and "replace" the batteries installed in the bicycles, as actions related to the relocation of bicycles.

Assuming a shared transportation service as described above, the learning and inference unit 12 uses impressions, which are the number of times an advertisement can be viewed, as one element of the "reward" included in the value of an action, and calculates (a) a placement evaluation value for placement, (b) a recovery evaluation value for recovery, and (c) a battery replacement evaluation value for battery replacement for each port using a calculation formula described below, and performs reinforcement learning based on the (a) placement evaluation value, (b) recovery evaluation value, and (c) battery replacement evaluation value obtained for all ports.

In addition, in the "stage of inferring an action" after reinforcement learning, the learning inference unit 12 has the function of substituting the current state information acquired by the acquisition unit 11 into the action value function obtained by reinforcement learning to acquire the action value when various actions are performed in the current state, and selecting the action that maximizes the acquired action value as the action to be taken.

The "learning stage processing" related to the above reinforcement learning and the "inference stage processing" related to behavior inference will be described later using Figures 12 to 14.

(Supplementary explanation of reinforcement learning overview and learning method (Figures 2 to 6))
As mentioned above, reinforcement learning is known as one type of machine learning, which deals with the problem of an agent in an environment observing the current state and determining the future action to be taken based on a policy from the state information obtained from the observation. The agent obtains a reward from the environment by executing the determined action, and reinforcement learning learns a policy that will obtain the most reward through a series of actions.

The applicant focuses on the above-mentioned reinforcement learning, and below discloses a technology that utilizes a reinforcement learning framework to optimize vehicle reallocation between multiple ports in a shared transportation service so as to maximize the reward in reinforcement learning.

As shown in FIG. 2, a "state" in reinforcement learning is information that an agent obtains from the environment, such as the current port state, demand forecast results, weather, distance, and restricted ports. An "action" is a behavior that an agent takes in the environment, such as placement, recovery, battery replacement, and which port to go to. A "reward" is a profit that an agent obtains in the environment, such as the "discounted cumulative reward R _t ," which will be described later with reference to FIG. 4. A "policy" is a function π that returns an action from a state.

As shown in the lower part of Figure 2, when state S _t obtained from the environment is input to function π related to the policy, action a _t is returned. Therefore, when the agent performs action a _t in state S _t , it reaches state S _t+1 . The probability of transitioning to state S _t+1 (state transition probability) is expressed as T(S _t+1 | S _t , a _t ), and the reward when the agent performs action a _t in state S _t is expressed as reward function R(S _t , a _t ).

As shown in Fig. 3, examples of information for states S _t and S _t+1 include port status, port demand, weather, truck status, bicycle status, port location, and distance between ports. By inputting state S _t into function π related to the policy, as action a _t , for example, for "truck A" as an agent, actions such as "port A" as the "next port,""-23" as the number of bicycles placed/recovered, and "10" as the number of batteries exchanged are returned. Note that when the number of bicycles placed/recovered is a positive value, it means that the number of bicycles will be placed, and when it is a negative value, it means that the number of bicycles will be recovered, so the above example means that 23 bicycles will be recovered at the next port A and 10 batteries will be replaced. In state S _t, an agent (truck A) performs action a _t to reach state S _t+1 , and by inputting state S _t+1 into function π related to the policy, as action a _t+1 , for example, for "truck A" as an agent, actions such as "port C" as the "next port,""10" as the number of bicycles placed/collected, and "0" as the number of batteries replaced are returned. This action means that 10 bicycles will be placed at the next port C, and no battery replacement will be performed (no replacement is required).

ここで、γ（０≦γ＜１）は割引率と呼ばれる定数である。つまり、将来の報酬を加味する必要があるが、実際には、将来の報酬は現時点では分からないため、上記のような割引率γを想定する。このとき、以下のように、将来へ向けて１世代ごとに割引率γを乗算していく。即ち、
１世代後の報酬R(S_t+1, a_t+1)は、γ¹R(S_t+1, a_t+1)
２世代後の報酬R(S_t+2, a_t+2)は、γ²R(S_t+2, a_t+2)
３世代後の報酬R(S_t+3, a_t+3)は、γ³R(S_t+3, a_t+3)
となり、ここで、
割引累積報酬R_t
＝γ¹R(S_t+1, a_t+1)＋γ¹R(S_t+1, a_t+1)＋γ¹R(S_t+1, a_t+1)＋γ¹R(S_t+1, a_t+1)＋…
と表され、図４の最下行にて破線で囲んだ部分はγR_t+1に相当するため、上記式（１）が導かれる。 As shown in FIG. 4, in this case, the “reward” in reinforcement learning is defined as the “discounted cumulative reward Rt” as shown in the following equation (1).

Here, γ (0≦γ<1) is a constant called the discount rate. In other words, it is necessary to take future rewards into account, but in reality, future rewards are unknown at the present time, so the above discount rate γ is assumed. In this case, the discount rate γ is multiplied for each generation going into the future as follows. In other words,
The reward R(S _t+1 , a _t+1 ) after one generation is γ ¹ R(S _t+1 , a _t+1 )
The reward R(S _t+2 , a _t+2 ) after two generations is γ ² R(S _t+2 , a _t+2 )
The reward R(S _t+3 , a _t+3 ) after three generations is γ ³ R(S _t+3 , a _t+3 )
Here,
Discounted cumulative reward R _t
=γ ¹ R(S _t+1 , a _t+1 )＋γ ¹ R(S _t+1 , a _t+1 )＋γ ¹ R(S _t+1 , a _t+1 )＋γ ¹ R(S _t+1 , a _t+1 )＋…
Since the part enclosed by the dashed line in the bottom row of FIG. 4 corresponds to γR _t+1 , the above formula (1) is derived.

上記式（２）の２段目の後段（図５にて破線で囲んだ部分）が行動価値関数Q(S_t, a_t)に相当する。このとき、行動価値関数を正確に見積もることができれば、効率よく報酬を得ることができる。 In addition, as shown in Fig. 5, in reinforcement learning, in order to learn a better policy, a state value function that estimates the value of a state and an action value function that estimates the value of an action are defined. Here, the "state value function V _π (S _t )" is a function that indicates how much discounted reward can be obtained in the future starting from the state S _t if the policy π is followed. The "action value function" is a function that indicates how much discounted reward can be obtained in the future if a certain action is taken from the state S _t , and constitutes a part of the above state value function.

The latter part of the second stage of the above formula (2) (the part surrounded by the dashed line in FIG. 5) corresponds to the action value function Q(S _t , a _t ). In this case, if the action value function can be accurately estimated, rewards can be obtained efficiently.

を最小化するように行動価値関数Q(S_t, a_t)を学習する。具体的には、

を最小化するように行動価値関数Q(S_t, a_t)を学習する。 Therefore, as shown in Figure 6, the error between the estimated value and the empirical value

The action value function Q(S _t , a _t ) is learned so as to minimize

We learn the action value function Q(S _t , a _t ) to minimize

(Explanation of Reward Calculation Example in Reinforcement Learning (FIGS. 7 to 11))
As described above, the learning and inference unit 12 uses the impression, which is the number of times an advertisement can be viewed, as one element of the "reward" included in the value of the action, and calculates for each port (a) a placement evaluation value related to placement, (b) a recovery evaluation value related to recovery, and (c) a battery replacement evaluation value related to battery replacement using a calculation formula described later, and performs reinforcement learning based on the (a) placement evaluation value, (b) recovery evaluation value, and (c) battery replacement evaluation value obtained for all ports. The calculation of the (a) placement evaluation value, (b) recovery evaluation value, and (c) battery replacement evaluation value will be described below with reference to Figures 7 to 11.

(a) The placement evaluation value, as shown in FIG.
Placement evaluation value = (average number of times bicycles are used when placed at the port x usage fee x percentage of members who use the bicycle once x constant α + average number of impressions of the bicycle x probability of seeing the bicycle x advertising cost x constant β) x number of bicycles to be placed (5)
It is calculated by the following formula (5).

(b) The recovery evaluation value is as shown in FIG.
Recovery evaluation value = ((Maximum number of average times used by placing it at another port - Average number of times used by placing it at the port) x Usage fee x Percentage of one-time members x Constant α - (Maximum number of average times of impressions of bicycles at other ports - Average number of impressions of bicycles) x Bicycle visibility probability x Advertising cost x Constant β) x Number of bicycles to be recovered (6)
It is calculated by the following formula (6).

(c) The battery exchange evaluation value, as shown in FIG.
Battery replacement evaluation value = (average number of times the bicycle is used when placed at the port x usage fee x percentage of members who use the bicycle once x constant α) x number of bicycles to replace the battery (7)
This is calculated by the following formula (7).

Next, each element of the formulas (5) to (7) will be outlined.
The "average number of times a device is used by being placed in a port" will be described later with reference to FIG. 8 and FIG.
The "average number of impressions for a bicycle" will be described later with reference to FIG. 10 and FIG.
"Usage fee" is the usage fee for the electric bicycle sharing service, for example 165 yen.
- "One-time member ratio" is the ratio of one-time members to the total number of members. Here, there are two types of members: monthly members who pay a fixed monthly amount, and one-time members who pay the above-mentioned usage fee each time they use the service. The one-time member ratio is calculated by the number of one-time members / (number of one-time members + number of monthly members).
"Advertising cost" is the cost per impression of a DressGuard advertisement placed on the body of a shared bicycle.
- "Constants α, β" are constants for adjusting the importance, which are determined taking into account the profit obtained by placing a bicycle at a port and the profit obtained by dressguard advertising, and the default is set to "1".
- "Bicycle visibility probability" is the probability that people who are estimated to have actually viewed the advertisement among bicycle impressions.
"The maximum average number of times used by placing it in another port" is the maximum value of "the average number of times used by placing it in a port" which will be described later.
"Maximum number of average impressions of bicycles of other ports" is the maximum value of "Average number of impressions of bicycles" described later.
"The number of bicycles to place,""The number of bicycles to retrieve," and "The number of bicycles to replace the batteries" are information contained in the action a that is returned by inputting the state S into the policy π.

As shown in FIG. 8, the "average number of times used by leaving it at a port" is calculated by the following procedure: counting the number of times each bicycle is used from a certain time (reference time) until it is collected by a truck or the battery is replaced (step 1); and calculating the average number of times the bicycle is used by weekday/holiday, time, port, and weather (step 2). FIG. 8 illustrates an example of a situation in which, after the reference time of 12:00, bicycle A is used from port A to port B, then from port B to port C, then from port C to port D, and then from port D to port E, and then collected by a truck at port E. In this situation, bicycle A's battery has not been replaced since the reference time of 12:00, and bicycle A has been used four times before it is collected by a truck, so in step 1, bicycle A is counted as "number of times used: 4 times." This counting is performed for each bicycle. Furthermore, in step 2, the average number of times the bicycle is used is calculated by weekday/holiday, time, port, and weather, as shown in the table in the lower right of FIG. 8. At this time, the weather is recorded as "sunny" if the hourly precipitation is 0 mm, and "rainy" if the hourly precipitation is > 0 mm. Also, the "port" in the summary table is the port where the bicycle was first used after the reference time.

More specifically, as shown in FIG. 9, the acquisition unit 11 extracts the usage history information for each bicycle from the usage table that stores the bicycle usage history information, and obtains the usage start date/time (reference time) and number of uses from the usage history information for bicycle A that is the subject of the calculation, as shown in FIG. 8. The acquisition unit 11 then references the weather information at the usage start date/time to determine the weather at that time, and calculates the average number of times the bicycle is used for weekdays/holidays, time, port, and weather, as shown in the table in the lower right of FIG. 9.

As shown in Figure 10, the "average number of impressions for a bicycle" is calculated by tallying up the number of times an advertisement is likely to be viewed for each bicycle from a certain time (reference time) until the bicycle is collected by a truck or the battery is replaced (step 1), as described below, and then calculating the average number of impressions for the bicycle for weekdays/holidays, time, port, and weather (step 2). Note that the weather is determined in the same way as in the example of Figure 8 described above.

More specifically, as shown in FIG. 11, the acquisition unit 11 extracts usage history information for each bicycle from a usage table that stores bicycle usage history information. The acquisition unit 11 also extracts people who are within a certain range of the location of the target bicycle (bicycle A in FIG. 11) from the bicycle location information and person location information, and counts up the number of people, using the obtained number as the number of impressions (the number of times the advertisement can be viewed) for the target bicycle A. This number of impressions is calculated for each bicycle usage history and merged with the usage history information for each bicycle extracted above, to obtain the number of impressions for each bicycle for each usage start date and time (reference time). The acquisition unit 11 then refers to the weather information at the usage start date and time (reference time) to determine the weather at that time, and calculates the average number of impressions for the bicycle for weekdays/holidays, time, port, and weather, as shown in the table in the lower right of FIG. 11.

(Explanation of the learning stage process (FIGS. 12-13))
The "learning stage processing" and the "inference stage processing" executed in the information processing device 10 will be described below in this order.

12 shows the process flow of the learning stage. First, the acquisition unit 11 extracts the time from a certain time until collection and the time from a certain time until battery replacement for each bicycle from the bicycle usage table of FIG. 9 and FIG. 11 (step S1). The above "certain time" may be selected from a number of predetermined reference time candidates. Next, the acquisition unit 11 calculates the average number of times the bicycle is used by being placed in a port using the procedure described above with reference to FIG. 8 and FIG. 9 (step S2). As a result, the average number of times the bicycle is used is calculated for each weekday/holiday, time, port, and weather, as shown in the table at the bottom right of FIG. 9. Furthermore, the acquisition unit 11 calculates the average number of impressions using the procedure described above with reference to FIG. 10 and FIG. 11 (step S3). As a result, the average number of impressions of the bicycle is calculated for each weekday/holiday, time, port, and weather, as shown in the table at the bottom right of FIG. 11.

となる行動a_kが求められる。さらに、学習推論部１２は、ステップＳ４Ｄで得られた行動a_kを用いてシミュレーションを行うことで、次の状態S_t+1を取得し（ステップＳ４Ｅ）、以上のようにして得られた、状態S_t、行動a_t（即ち、割引累積報酬が最大となる行動a_k）、報酬R(S_t, a_t)（即ち、状態S_tから行動a_kを実行した場合の割引累積報酬）、および、次の状態S_t+1から成る情報(S_t, a_t, R(S_t, a_t), S_t+1)を蓄積する（ステップＳ４Ｆ）。 Next, the learning inference unit 12 executes the accumulation process of information (S _t , a _{t ,} R(S _t , a _t ), S _t+1 ) shown in FIG. 13 (step S4). First, the learning inference unit 12 acquires the state S _t acquired by the acquisition unit 11 (step S4A in FIG. 13), and obtains an action a _t by inputting the acquired state S _t to the policy π (step S4B). At this time, in reality, multiple possible actions a ₁ , a ₂ , ..., _an are obtained. Then, the learning inference unit 12 calculates the discounted cumulative reward R(S t , a _t ) when each of the actions a _t (a ₁ , a ₂ , ..., _an ) is executed from the state _{S t} using the formula of the discounted cumulative reward Rt (step S4C). Specifically, the "placement evaluation value,""recovery evaluation value," and "battery exchange evaluation value" described with reference to Figures 7 to 11 are calculated for each bicycle, and the discounted cumulative reward R(S _t , a _t ) is calculated from the above three evaluation values for all bicycles. The learning and inference unit 12 then determines the action a _k that maximizes the calculated discounted cumulative reward (step S4D). That is, in step S4D, of the multiple possible actions a ₁ , a ₂ , ..., a _n ,

Further, the learning and inference unit 12 performs a simulation using the action a _k obtained in step S4D to obtain the next state S _t+1 (step S4E), and accumulates information (S t , a t , R(S _t , a _{t ), S t+1 ) consisting of the state S t , action a t (i.e., the action a k with the largest discounted cumulative reward), reward R(S t , a t ) (i.e., the discounted cumulative reward when the action a k is executed from the state S t ) , and the next state S t + 1 obtained as} described above (step S4F).

を最小化するように行動価値関数Q(S_t, a_t)を学習する（ステップＳ５）。その後、学習推論部１２は、ステップＳ４、Ｓ５の処理を所定のループ回数だけ繰り返すことで、行動価値関数Q(S_t, a_t)を学習し、そして、ループ回数繰り返した後、図１２の処理を終了する。 Returning to FIG. 12, the learning and inference unit 12

(Step S5). After that, the learning and inference unit 12 repeats the processes of steps S4 and S5 a predetermined number of times to learn the action value function _{Q(S t ,} a _t ). After the loop is repeated a predetermined number of times, the process of FIG _. 12 ends.

(Explanation of the inference stage process (FIG. 14))
FIG. 14 shows a process flow of the inference stage. First, the acquisition unit 11 acquires state information S _t in the same manner as the learning stage process described above (step S11), and the learning inference unit 12 calculates the action value for each action a _t using the action value function Q(S _t , a _t ) obtained by learning (step S12). Then, the learning inference unit 12 selects the action a _t with the maximum action value (step S13). In this way, the action with the highest value of future action is selected.

(Effects of the above embodiment)
The effects of the above embodiment will be described below.

By using the above-mentioned reinforcement learning, the action value function is trained to an action value function that selects the action that will have the highest value for future actions. Based on this action value function, the value of actions related to vehicle reallocation between multiple ports is estimated, and control is exercised so that the action that will have the highest value for future actions is selected. Therefore, when reallocating vehicles between multiple ports in a shared transportation service, vehicle reallocation can be optimized to maximize the reward in reinforcement learning.

Specifically, the actions assumed for the reallocation of bicycles shared in a shared transportation service are "placement," "collection," and "battery replacement." Appropriate reinforcement learning is performed based on evaluation values from multiple aspects, namely placement, collection, and battery replacement, which allows for more appropriate optimization of vehicle reallocation and maximizes rewards.

The bicycles shared in the shared transportation service have advertisements displayed on the body of the bicycle, and the learning and inference unit 12 calculates the placement evaluation value and the collection evaluation value while taking into account impressions, which are the number of times an advertisement can be viewed, so that rewards from advertising can be maximized.

Although rental electric bicycles have been given as an example of vehicles shared in the shared transportation service, any vehicle that can be restarted by replacing the battery and has advertisements on the body can be used, and the present disclosure can also be applied to other types of vehicles such as rental electric kick scooters, rental motorcycles, and rental cars. In this way, the present disclosure can be applied to a wide range of vehicles, including rental electric kick scooters, which have become increasingly popular in recent years, and has the potential to be widely used by users.

In addition, at the stage of inferring actions, by executing the process in Figure 14, the action that maximizes the action value can be selected as the action to be taken, thereby maximizing the reward (profit) for the service.

Note that the above embodiment is one example of the present disclosure, and the type of information, the content of the reward, the content of the processing, etc. are not limited to those described above.

The gist of the present disclosure lies in the following [1] to [5].
[1] An acquisition unit that acquires information to be input into an action value function that estimates the value of an action related to the reallocation of vehicles among a plurality of ports in a sharing transportation service in which vehicles are shared;
a learning and inference unit that performs reinforcement learning on the action-value function using the information acquired by the acquisition unit in order to make the action-value function select an action that will have the highest value for a future action;
An information processing device comprising:
[2] A vehicle shared in the shared transportation service is a vehicle that can be reused by replacing the installed battery,
The actions related to the relocation of the vehicle include the deployment of the vehicle to a port, the recovery of the vehicle from a port, and the replacement of a battery installed in the vehicle;
The learning and inference unit uses a predetermined calculation formula to calculate a placement evaluation value for the placement, a recovery evaluation value for the recovery, and a battery replacement evaluation value for the battery replacement for each port as rewards included in the value of the action, and performs the reinforcement learning based on the placement evaluation values, recovery evaluation values, and battery replacement evaluation values obtained for all ports.
[3] A vehicle shared in the sharing transportation service is a vehicle on which advertisements are displayed,
The information processing device according to [2], wherein the learning and inference unit calculates the placement evaluation value and the collection evaluation value using impressions, which are the number of times an advertisement can be viewed, as one element.
[4] The information processing device described in [3], wherein the vehicles shared in the shared transportation service are rental electric bicycles, rental electric kick scooters, rental motorbikes, and rental cars that can be restarted by replacing the batteries and have advertisements displayed on the vehicle bodies.
[5] In the step of inferring an action, the learning inference unit
The information processing device according to any one of [1] to [4], wherein the current state information acquired by the acquisition unit is substituted into the action value function obtained by the reinforcement learning to obtain action values for various actions taken in the current state, and the action that maximizes the obtained action value is selected as the action to be taken.

(Explanation of terms, explanation of hardware configuration (Fig. 15), etc.)
The block diagrams used in the description of the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. The method of realizing each functional block is not particularly limited. That is, each functional block may be realized using one device that is physically or logically coupled, or may be realized using two or more devices that are physically or logically separated and directly or indirectly connected (for example, using wires, wirelessly, etc.). The functional blocks may be realized by combining the one device or the multiple devices with software.

Functions include, but are not limited to, judgement, determination, judgment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, regarding, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assignment. For example, a functional block (component) that performs the transmission function is called a transmitting unit or a transmitter. As mentioned above, there are no particular limitations on the method of realization for either of these.

For example, an information processing device in an embodiment of the present disclosure may function as a computer that executes the processing of the present disclosure. FIG. 15 is a diagram showing an example of the hardware configuration of an information processing device 10 according to an embodiment of the present disclosure. The information processing device 10 described above may be physically configured as a computer device including a processor 1001, memory 1002, storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, etc.

In the following description, the word "apparatus" can be interpreted as a circuit, device, unit, etc. The hardware configuration of the information processing device 10 may be configured to include one or more of the devices shown in the figure, or may be configured to exclude some of the devices.

Each function of the information processing device 10 is realized by loading a specific software (program) onto hardware such as the processor 1001 and memory 1002, causing the processor 1001 to perform calculations, control communications via the communication device 1004, and control at least one of the reading and writing of data in the memory 1002 and storage 1003.

The processor 1001, for example, runs an operating system to control the entire computer. The processor 1001 may be configured as a central processing unit (CPU) that includes an interface with peripheral devices, a control device, an arithmetic unit, registers, etc.

The processor 1001 also reads out programs (program codes), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002, and executes various processes according to these. The programs used are those that cause a computer to execute at least some of the operations described in the above-mentioned embodiments. Although it has been described that the various processes are executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The programs may be transmitted from a network via a telecommunications line.

Memory 1002 is a computer-readable recording medium, and may be composed of at least one of, for example, ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), etc. Memory 1002 may also be called a register, cache, main memory (primary storage device), etc. Memory 1002 can store executable programs (program codes), software modules, etc. for implementing a wireless communication method according to one embodiment of the present disclosure.

Storage 1003 is a computer-readable recording medium, and may be, for example, at least one of an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. Storage 1003 may also be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other suitable medium including at least one of memory 1002 and storage 1003.

The communication device 1004 is hardware (transmitting/receiving device) for communicating between computers via at least one of a wired network and a wireless network, and is also referred to as, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 1004 may be configured to include a high-frequency switch, a duplexer, a filter, a frequency synthesizer, etc., to realize, for example, at least one of Frequency Division Duplex (FDD) and Time Division Duplex (TDD).

The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that performs output to the outside. Note that the input device 1005 and the output device 1006 may be integrated into one structure (e.g., a touch panel).

Furthermore, each device such as the processor 1001 and memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

In addition, the information processing device 10 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.

The notification of information is not limited to the aspects/embodiments described in this disclosure, and may be performed using other methods. For example, the notification of information may be performed by physical layer signaling (e.g., DCI (Downlink Control Information), UCI (Uplink Control Information)), higher layer signaling (e.g., RRC (Radio Resource Control) signaling, MAC (Medium Access Control) signaling, broadcast information (MIB (Master Information Block), SIB (System Information Block))), other signals, or a combination of these. In addition, RRC signaling may be referred to as an RRC message, and may be, for example, an RRC Connection Setup message, an RRC Connection Reconfiguration message, etc.

Each aspect/embodiment described in this disclosure is a mobile communication system that is compatible with LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), 6th generation mobile communication system (6G), xth generation mobile communication system (xG) (xG (x is, for example, an integer or decimal number)), FRA (Future Ra The present invention may be applied to at least one of systems using IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth (registered trademark), and other appropriate systems, and next-generation systems that are expanded, modified, created, or defined based on these. It may also be applied to a combination of multiple systems (for example, a combination of at least one of LTE and LTE-A with 5G, etc.).

The processing steps, sequences, flow charts, etc. of each aspect/embodiment described in this disclosure may be reordered unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order and are not limited to the particular order presented.

The input and output information may be stored in a specific location (e.g., memory) or may be managed using a management table. The input and output information may be overwritten, updated, or added to. The output information may be deleted. The input information may be sent to another device.

The determination may be based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., a comparison with a predetermined value).

Each aspect/embodiment described in this disclosure may be used alone, in combination, or switched depending on the execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information).

　Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.

Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

Software, instructions, information, etc. may also be transmitted and received via a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using at least one of wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and/or wireless technologies (such as infrared, microwave), then at least one of these wired and wireless technologies is included within the definition of a transmission medium.

The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

Note that the terms explained in this disclosure and the terms necessary for understanding this disclosure may be replaced with terms having the same or similar meanings. For example, at least one of the channel and the symbol may be a signal (signaling). Also, the signal may be a message. Also, the component carrier (CC) may be called a carrier frequency, a cell, a frequency carrier, etc.

As used in this disclosure, the terms "system" and "network" are used interchangeably.

In addition, the information, parameters, etc. described in this disclosure may be represented using absolute values, may be represented using relative values from a predetermined value, or may be represented using other corresponding information. For example, a radio resource may be indicated by an index.

The names used for the parameters described above are not intended to be limiting in any way. Furthermore, the formulas etc. using these parameters may differ from those explicitly disclosed in this disclosure. The various channels (e.g., PUCCH, PDCCH, etc.) and information elements may be identified by any suitable names, and therefore the various names assigned to these various channels and information elements are not intended to be limiting in any way.

As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in a table, database, or other data structure), and considering ascertaining as "judging" or "determining." Also, "determining" and "determining" may include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and considering ascertaining as "judging" or "determining." Additionally, "judgment" and "decision" can include considering resolving, selecting, choosing, establishing, comparing, etc., to have been "judged" or "decided." In other words, "judgment" and "decision" can include considering some action to have been "judged" or "decided." Additionally, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

Any reference to an element using a designation such as "first," "second," etc., used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must precede the second element in some way.

When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Additionally, the term "or," as used in this disclosure, is not intended to be an exclusive or.

In this disclosure, where articles have been added through translation, such as a, an, and the in English, this disclosure may include that the nouns following these articles are plural.

In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

10: Information processing device, 11: Acquisition unit, 12: Learning and inference unit, 1001: Processor, 1002: Memory, 1003: Storage, 1004: Communication device, 1005: Input device, 1006: Output device, 1007: Bus.

Claims (5)

An acquisition unit that acquires information to be input to an action value function that estimates the value of an action related to the reallocation of vehicles among a plurality of ports in a shared transportation service that shares vehicles;
a learning and inference unit that performs reinforcement learning on the action-value function using the information acquired by the acquisition unit in order to make the action-value function select an action that will have the highest value for a future action;
An information processing device comprising: The vehicle shared in the shared transportation service is a vehicle that can be reused by replacing the installed battery,
The actions related to the relocation of the vehicle include the deployment of the vehicle to a port, the recovery of the vehicle from a port, and the replacement of a battery installed in the vehicle;
the learning and inference unit calculates, for each port, a placement evaluation value related to the placement, a recovery evaluation value related to the recovery, and a battery replacement evaluation value related to the battery replacement, using a predetermined calculation formula as a reward included in the value of the action, and performs the reinforcement learning based on the placement evaluation values, recovery evaluation values, and battery replacement evaluation values obtained for all ports.
The information processing device according to claim 1 . The vehicle shared in the shared transportation service is a vehicle on which advertisements are displayed,
the learning and inference unit calculates the placement evaluation value and the recovery evaluation value using, as one element, impressions, which are the number of times an advertisement can be viewed;
The information processing device according to claim 2 . The vehicles shared in the shared transportation service are rental electric bicycles, rental electric kick scooters, rental motorcycles, and rental cars that can be restarted by replacing the batteries and have advertisements placed on the vehicle body.
The information processing device according to claim 3 . In the step of inferring an action, the learning inference unit
The current state information acquired by the acquisition unit is substituted into the action value function obtained by the reinforcement learning to acquire action values in the case where various actions are performed in the current state, and the action that maximizes the acquired action value is selected as the action to be taken.
The information processing device according to claim 1 .

Images