Improving reinforcement learning algorithms: towards optimal learning rate policies
share
This paper investigates to what extent we can improve reinforcement learning algorithms. Our study is split in three parts. First, our analysis shows that the classical asymptotic convergence rate O(1/√(N)) is pessimistic and can be replaced by O((log(N)/N)^β) with 1/2≤β≤ 1 and N the number of iterations. Second, we propose a dynamic optimal policy for the choice of the learning rate (γ_k)_k≥ 0 used in stochastic algorithms. We decompose our policy into two interacting levels: the inner and the outer level. In the inner level, we present the PASS algorithm (for "PAst Sign Search") which, based on a predefined sequence (γ^o_k)_k≥ 0, constructs a new sequence (γ^i_k)_k≥ 0 whose error decreases faster. In the outer level, we propose an optimal methodology for the selection of the predefined sequence (γ^o_k)_k≥ 0. Third, we show empirically that our selection methodology of the learning rate outperforms significantly standard algorithms used in reinforcement learning (RL) in the three following applications: the estimation of a drift, the optimal placement of limit orders and the optimal execution of large number of shares.
READ FULL TEXT