Abstract: In the field of deep reinforcement learning， in order to further reduce the impact of value
overestimation on policy estimation in TD3 ( Twin Delayed Deep Deterministic Policy Gradients) and accelerate
the efficiency of model learning，a DD-TD3 ( Twin Delayed Deep Deterministic Policy Gradients with Dynamic
Delayed Policy Update) is proposed． The delay update step size of the actor network is guided by the dynamic
difference between the latest loss of the critic network and its exponential weighted moving average． Experimental
results show that compared to the original TD3 algorithm that obtain high reward value in the 2 000 steps，the
DD-TD3 method can learn the optimal control strategy in about 1 000 steps and obtain a higher reward value，
thereby the efficiency of finding the optimal strategy is improved．

Key words: deep reinforcement learning, twin delayed deep deterministic policy gradients ( TD3) , dynamic delayed policy update