Dfyjkt

Hello Dear StackOverflow Community!

I am taking a Reinforcement Learning course right now and have a confusion about the Policy Iteration method.

In the Policy Iteration, we have a random policy at the beginning and value functions for each state. In the "Policy Evaluation" part of the Policy Iteration process, we find new value functions for each state. After that, in the "Policy Improvement" part, based on the new value functions, we update our policy. We do these steps iteratively until our value functions converge. But my problem is, how do we adapt our new policy for the next policy evaluation process? Let me explain my point on an example.

Grid world example:

As you can see on the image, black boxes are terminal states, our immediate reward is -1, discount factor is 1, and 0.25 possibility for all directions at the beginning. In the policy we get at the policy improvement part k = 1, we have to go left from the state which is on the right of the top-leftmost terminal state. After that, while updating the value functions for k = 2, why don't we consider this change and write -1.75 (abbreviated as -1.7) to this state like we can go all directions? In my opinion as mathematically:

new value function for this state = -1 + 1 * (0.0) = -1

This should be like that because we have only one possible action from that state based on our last policy instead of:

-1.7 = ((-1) + 1 * (-1)) * (0.75) + (-1 + 1 * (0.0)) * (0.25)

If we do it like that, so what is the intermediate missions of those policies? Just do it as value iteration if we do not use them for the new value functions!

I am so confused about that, can you help me please?