机器学习（十） — 强化学习

Reinforcement learning

1 key concepts

1. states
2. actions
3. rewards
4. discount factor $\gamma$
5. return
6. policy $\pi$

2 return

1. definition: the sum of the rewards that the system gets, weighted by the discount factor
2. compute:
   • $R_i$ : reward of state i
   • $\gamma$ : discount factor(usually close to 1), making the reinforcement learning impatient

$$return=R_1+\gamma R_2+\cdots +{\gamma }^{n-1}{R}_n$$

3 policy

policy $\pi$ maps state $s$ to some action $a$

$$\pi (s)=a$$

the goal of reinforcement learning is to find a policy $\pi$ to map every state $s$ to action $a$ to maximize the return

4 state action value function

1. definition

$Q(s, a) = $return if

• start in state $s$
• take action $a$ once
• behave optimally after that

2. usage

1. the best possible return from state $s$ is $max$ $Q(s,a)$
2. the best possible action in state $s$ is the action $a$ that gives $max$ $Q(s,a)$

5 bellman equation

$s$ : current state

$a$ : current action

$s^{{}^{\prime }}$ : state you get to after taking action $a$

$a^{{}^{\prime }}$ : action that you take in state $s^{{}^{\prime }}$

$$Q(s,a)=R(s)+\gamma maxQ(s^{{}^{\prime }},a^{{}^{\prime }})$$

6 Deep Q-Network

1. definition

use neural network to learn $Q(s,a)$

$$\begin{gathered} x=(s,a) \\ y=R(s)+\gamma maxQ(s^{{}^{\prime }},a^{{}^{\prime }}) \\ {f}_{w,b}(x)\approx y \end{gathered}$$

2. step

1. initialize neural network randomly as guess of $Q(s,a)$
2. repeat:
   • take actions, get $(s,a,R(s),s^{{}^{\prime }})$
   • store N most recent $(s,a,R(s),s^{{}^{\prime }})$ tuples
3. train neural network:
   • create training set of N examples using $x=(s,a)$ and $y=R(s)+\gamma maxQ(s^{{}^{\prime }},a^{{}^{\prime }})$
   • train $Q_{new}$ such that $Q_{new}\approx y$
   • set $Q=Q_{new}$

3. optimazation

4. $ϵ$-greedy policy

1. with probability $1-ϵ$, pick the action $a$ that maximize $Q(s,a)$
2. with probability $ϵ$, pick the action $a$ randomly

5. mini-batch

use a subset of the dataset on each gradient decent

6. soft update

instead $Q=Q_{new}$

$$\begin{gathered} w=\alpha w_{new}+w \\ b=\alpha b_{new}+b \end{gathered}$$