Abstract: To solve the problem that Q function overestimation may cause SAC ( Soft Actor Critic) algorithm trapped in local optimal solution, SCSAC ( Stable Constrained Soft Actor Critic) algorithm is proposed for perfectly resolving the above weakness hidden in maximum entropy objective function improving the stability of Stable Constrained Soft Actor Critic algorithm in trailing process. The result of evaluating Stable Constrained Soft Actor Critic algorithm on the suite of OpenAI Gym Mujoco environments shows less Q value overestimation appearance and more stable results in trailing process comparing with SAC algorithm.

Key words: reinforcement learning, maximum entropy reinforcement learning, Q value overestimation, soft actor critic(SAC)algorithm