CN113339499A

CN113339499A - Intelligent gear shifting rule control method based on Q-Learning reinforcement Learning algorithm

Info

Publication number: CN113339499A
Application number: CN202110760756.6A
Authority: CN
Inventors: 张坤; 郭洪强; 崔庆虎; 赵峰睿
Original assignee: Liaocheng University
Current assignee: Liaocheng University
Priority date: 2021-07-04
Filing date: 2021-07-04
Publication date: 2021-09-03
Anticipated expiration: 2041-07-04
Also published as: CN113339499B

Abstract

The invention discloses an intelligent gear shifting rule control method based on a Q-Learning reinforcement Learning algorithm, which controls the lifting of gears through the reinforcement Learning algorithm, enables an AMT (automated mechanical transmission) to always work under the gear with the best energy consumption and economy through repeated training, solves the problem that self-adaptive gear shifting control cannot be realized under a dynamic unknown working condition, further reduces the energy consumption, and further improves the dynamic property and the economy of the whole vehicle.

Description

Intelligent gear shifting rule control method based on Q-Learning reinforcement Learning algorithm

Technical Field

The invention relates to the field of AMT intelligent control, in particular to an intelligent gear shifting rule control method based on a Q-Learning reinforcement Learning algorithm.

Background

A large number of researches show that the power performance and the economical efficiency of the whole vehicle are greatly influenced by the gear shifting rule, and the traditional regular gear shifting rule cannot adapt to dynamic and uncertain traffic environments due to parameter determination and cannot realize dynamic self-adaptive gear shifting control for adapting to different road traffic conditions, so that the power performance and the economical efficiency of the whole vehicle are difficult to achieve the optimal performance. Therefore, an intelligent gear shifting rule control method based on reinforcement learning is provided.

Disclosure of Invention

The invention aims to solve the technical problem of providing an intelligent gear shifting law control method based on a Q-Learning reinforcement Learning algorithm, which is suitable for dynamic self-adaptive gear shifting control under different road traffic conditions, so that the dynamic property and the economical efficiency of the whole vehicle are difficult to achieve the optimal property.

In order to solve the technical problems, the invention adopts the following technical means:

an intelligent gear shifting rule control method based on a Q-Learning reinforcement Learning algorithm comprises the following steps:

p1: setting an action set A ═ a of reinforcement learning₁,a₂…a_tAnd state set S ═ S₁,s₂…s_t}; for action set a ═ a₁,a₂…a_tAn action value a (t) at time t indicates a target shift position to be selected at time t, that is, a (t) {1,2,3,4}, and a state set S { S ═ S }₁,s₂…s_tState s (t) at time t includes vehicle speed, acceleration and shift position at time t, i.e., s (t) ═ { v (t), acc (t), gear (t);

p2: respectively discretizing v and acc in the state set into 100 parts, wherein gear is 4 gears, namely, the total state variables in the state set are 100 multiplied by 4 to 40000, and then realizing state space reduction through the optimal Latin hypercube design;

p3: initializing a Q table constructed by a state set S and an action set A into an all-zero matrix;

p4: selecting an action value a (t) by an epsilon-greedy strategy according to a state s (t) at the time t in the environment;

p5: performing gear shifting according to the selected action value a (t), calculating a corresponding gear and motor efficiency, and acquiring a return value r (s (t) and a (t)) of the action value a (t) selected at the time t;

p6: and updating the Q table according to a Q learning formula, wherein the current environment is in the next state, and the updating formula is as follows:

wherein eta represents learning rate, 0< eta < 1, gamma represents discount factor, and 0< gamma < 1;

p7: the execution of P4 to P6 is repeated until the current environment is finished, i.e. the updating of the Q table is completed once.

The further preferred technical scheme is as follows:

in step P2, based on the memory space limitation of the controller, the number of state variables is reduced from 40000 to 1000 by implementing the state space reduction through the optimal latin hypercube design, which is beneficial to the practical application of the algorithm to the whole vehicle controller.

In step P4, an epsilon-greedy strategy is used to select actions, and the search factor is adjusted continuously according to the iteration of the training times, so that the closer the training times are to the end, the smaller the search probability and the larger the probability of selecting from the learned Q-table.

In step P5: after an action value a (t) is selected through an epsilon-greedy strategy, the action value a (t) is used as a gear action to obtain a target gear of the next second and a motor efficiency effective value of the next moment, and a return value r (s (t) under the action value a (t) at the moment t is obtained, wherein a (t) is defined as:

the efficiency value of the motor at the next moment after gear shifting is equal to the efficiency value of the motor at the next moment, the efficiency value of the motor at the current gear shifting is equal to the efficiency value of the motor at the next moment after gear shifting, if the efficiency value of the motor after gear shifting is high, a reward is given, and if the efficiency value of the motor after gear shifting is not high, a penalty is given.

Drawings

FIG. 1 is a schematic diagram of the effect of the optimal Latin hypercube algorithm.

Fig. 2 is a diagram of a reinforcement learning intelligent gear shift schematic.

FIG. 3 is a Q-learning reinforcement learning framework diagram.

Detailed Description

The present invention will be further described with reference to the following examples.

Referring to fig. 3, the intelligent gear shifting law control method based on the Q-Learning reinforcement Learning algorithm of the present invention includes the following steps:

p2: respectively discretizing v and acc in the state set into 100 parts, wherein gear is 4 gears, namely, the total state variables in the state set are 100 multiplied by 4 to 40000, and then realizing state space reduction through the optimal Latin hypercube design; as shown in fig. 1, an optimal latin hypercube design is introduced by state reduction, and a distance criterion is used as optimization judgment;

In step P2, based on the memory space limitation of the controller, the number of state variables is reduced from 40000 to 1000 by using the optimal latin hypercube design to reduce the number of state variables from 40000, which is beneficial to the practical application of the algorithm to the whole vehicle controller.

As shown in fig. 3, an Optimal Latin hypercube design (Opt LHD) can effectively improve sampling uniformity and stability, and can make the fitting between the factors and the response more accurate.

The calculation principle of Opt LHD can be explained as: assume that there is an n x m matrix, where n represents the test point and m represents the test factor. Row matrix X_i ^T＝[x_i1,x_i2,…,x_im]Representing the assay analysis and the column matrix representing the assay factors. Wherein, Opt LHD can judge whether to reach the optimum according to the distance criterion, the entropy criterion or the deviation criterion. This patent adopts the distance criterion to carry out optimization judgement.

The distance criterion can be calculated by the maximum and minimum values:

in the formula, d (x)_i,x_j) Represents two sample points (x)_i,x_j) The number of the state variables is reduced from 40000 to 1000 by the formula, so that the reduction of the state space is realized;

in step P4, an epsilon-greedy strategy is used to select actions, and the search factor is adjusted continuously according to the iteration of the training times, so that the closer the training times are to the end, the smaller the search probability and the larger the probability of selecting from the learned Q-table. Different from a random strategy and a greedy strategy, the method not only avoids iteration times increase caused by repeated action selection possibly generated by the random strategy, but also avoids the situation that the greedy strategy is locally optimal and lacks exploratory property.

The invention has the advantages that:

(1) by applying the optimal Latin hypercube design, the sampling uniformity and stability are effectively improved, and the fitting between the factors and the response is more accurate.

(2) The action value a (t) is selected through the epsilon-greedy strategy, so that the phenomenon that iteration times are increased due to repeated action selection possibly generated by a random strategy is avoided, and the phenomenon that a greedy strategy is locally optimal and lacks exploratory property is avoided.

(3) And (d) using the action value a (t) as a gear action to obtain a target gear for the next second and a motor efficiency value at the next moment, and obtaining a return value r (s (t), a (t)) under the action value a (t) at the moment t, so that the energy consumption is further reduced, and the energy-saving and environment-friendly effects are facilitated.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the scope of the present invention, which is defined in the appended claims.

Claims

1. An intelligent gear shifting rule control method based on a Q-Learning reinforcement Learning algorithm is characterized by comprising the following steps of:

p1: setting an action set A ═ a of reinforcement learning₁,a₂…a_tAnd state set S ═ S{s₁,s₂…s_t}; for action set a ═ a₁,a₂…a_tAn action value a (t) at time t indicates a target shift position to be selected at time t, that is, a (t) {1,2,3,4}, and a state set S { S ═ S }₁,s₂…s_tState s (t) at time t includes vehicle speed, acceleration and shift position at time t, i.e., s (t) ═ { v (t), acc (t), gear (t);

2. The intelligent gear shift schedule control method based on the Q-Learning reinforcement Learning algorithm as claimed in claim 1, wherein: in step P2, based on the memory space limitation of the controller, the number of state variables is reduced from 40000 to 1000 by implementing the state space reduction through the optimal latin hypercube design, which is beneficial to the practical application of the algorithm to the whole vehicle controller.

3. The intelligent gear shift schedule control method based on the Q-Learning reinforcement Learning algorithm as claimed in claim 1, wherein: in step P4, an epsilon-greedy strategy is used to select actions, and the search factor is adjusted continuously according to the iteration of the training times, so that the closer the training times are to the end, the smaller the search probability and the larger the probability of selecting from the learned Q-table.

4. The intelligent gear shift schedule control method based on the Q-Learning reinforcement Learning algorithm as claimed in claim 1, wherein: in step P5: after an action value a (t) is selected through an epsilon-greedy strategy, the action value a (t) is used as a gear action to obtain a target gear of the next second and a motor efficiency effective value of the next moment, and a return value r (s (t) under the action value a (t) at the moment t is obtained, wherein a (t) is defined as: