NmetaQ: An n-agent reinforcement learning algorithm based on meta equilibrium

Multi-agent reinforcement learning (MARL) has been widely studied over the last years. In MARL, one approach is to combine game theory with reinforcement learning (RL) to help with selecting actions and updating policies. Markov games are adopted in this approach as the framework and policies are learnt based on equilibrium theories. Several algorithms have been proposed based on this idea, such as minimax-Q, NashQ, FFQ, Correlated-Q and MetaQ. However, some of these algorithms are proposed only for 2-agent problems while the others have difficulty in dealing with problems with more than 2 agents. Since many tasks involve more than 2 agents in the real world, an algorithm which can deal with n-agent (n > 2) problems is needed. In this paper, we propose nMetaQ based on MetaQ. nMetaQ can be applied to a multi-agent environment that has more than 2 agents. Experimental results demonstrate the empirical convergence of nMetaQ and show its satisfactory adaptive performance. The most important advantage of nMetaQ is that it can work efficiently and effectively in an n-agent (n > 2) environment while previous methods may not.