1. Introduction
If we know all the probabilities of applying each possible action to each state, and thus the corresponding transitions, the problem we have to solve is how to search in an already known tree of states and actions the optimal action for each state.
2. Value & policy iteration
See more details here.