Foundations of Deep Reinforcement Learning: A review

The best book to learn Deep Reinforcement Learning

Foundations of Deep Reinforcement Learning: The following is a review of the book Foundations of Deep Reinforcement Learning: Theory and Practice in Python (Addison-Wesley Data & Analytics Series) by Laura Grasser and Wah Loon Keng, one of the best introductory books to the topic of Deep Reinforcement Learning.

Review

Reinforcement learning (RL) is the third and almost always forgotten type of learning in the Machine Learning world. It’s two brothers, Supervised and Un-Supervised learning are more widely known in general.

However, it is an amazingly exciting field that is populating the research centres of many top-notch universities and companies like Google. RL has awesome applications like guiding robitc arms, or teaching machines to play games that end up beating expert human players, like in the case of AlphaGo and Lee Sedol, or the famous Mar.io shown in the following video:

You can see a similar video with the reward function is displayed on real time here.

Foundations of Deep Reinforcement Learning is in our opinion the best book out there to get started on the topic. It provides an introduction to Deep RL that has both, greatly explained theory, and neat code implementations.

By the end of it you will know the theory and main concepts behind Deep Reinforcement Learning algorithms, how to implement them, as well the best practices and practical details of how to get RL to work.

You will get to know all of the following points:

• Understand each key aspect of a deep RL problem
• Explore policy- and value-based algorithms, including REINFORCE, SARSA, DQN, Double DQN, and Prioritized Experience Replay (PER)
• Delve into combined algorithms, including Actor-Critic and Proximal Policy Optimization (PPO)
• Understand how algorithms can be parallelized synchronously and asynchronously
• Run algorithms in SLM Lab and learn the practical implementation details for getting deep RL to work
• Explore algorithm benchmark results with tuned hyperparameters
• Understand how deep RL environments are designed

Book description and contents – All you need to know about Deep Reinforcement Learning

The book is a an introduction to deep reinforcement learning that covers all of the prevailing contemporary theory and applications. It is very practical, the theory in the chapters is easily readable, and there quick links, code and snippets available to try out on SLM lab.

The contents are the following:

• 1. Introduction to Reinforcement Learning

I: Policy-Based and Value-Based Algorithms

• 2. REINFORCE
• 3. SARSA
• 4. Deep Q-Networks (DQN)
• 5. Improving DQN
• 6. Advantage Actor-Critic (A2C)
• 7. Proximal Policy Optimization (PPO) 165
• 8. Parallelization Methods
• 9. Algorithm Summary

III: Practical Details

• 10. Getting Deep RL to Work 209
• 11. SLM Lab
• 12. Network Architectures
• 13. Hardware

IV: Environment Design

• 14. States
• 15. Actions
• 16. Rewards
• 17. Transition Function 333

Who is this book for?

Foundation of Deep Reinforcement Learning is oriented for people with software knowledge who have a basic understanding of Machine Learning and also know how to program in Python.

If you want to learn Deep RL, but don’t have this previous skills, don’t worry, we got you covered: our advice would be to follow the next steps: First pick a beginner Python book from our Python books section. Our favourite is probably Python Crash course.

Once you are comfortable with Python, I would suggest grabbing a Data analysis book, or take a course online to learn to use Pandas, Numpy and Matplotlib.

Lastly, get an introductory Machine Learning book, like The Hundred Page Machine Learning book, to get an idea of the main concepts of Machine Learning or follow an easy Machine Learning Course.

After this, you will be ready to tackle Foundations of Deep RL with ease.

Summary

Foundations of Deep Reinforcement Learning is a neatly written book, where the key concepts are clearly explained and easy to understand. We would definitely recommend this book to anyone that wants to grab the key ideas and start working on Deep RL right away.

While by itself it wont make you an expert, it will give you a fantastic foundation and set you up for more advanced material like the course on Udacity Deep Reinforcement Learning Nanodegree or other well known books on the subject like Reinforcement Learning, second edition: An Introduction (Adaptive Computation and Machine Learning series).

You can find the book on Amazon here:

Foundations of Deep Reinforcement Learning

Quotes about the book

“This book provides an accessible introduction to deep reinforcement learning covering the mathematical concepts behind popular algorithms as well as their practical implementation. I think the book will be a valuable resource for anyone looking to apply deep reinforcement learning in practice.”
–Volodymyr Mnih, lead developer of DQN

“An excellent book to quickly develop expertise in the theory, language, and practical implementation of deep reinforcement learning algorithms. A limpid exposition which uses familiar notation; all the most recent techniques explained with concise, readable code, and not a page wasted in irrelevant detours: it is the perfect way to develop a solid foundation on the topic.”
–Vincent Vanhoucke, principal scientist, Google

“As someone who spends their days trying to make deep reinforcement learning methods more useful for the general public, I can say that Laura and Keng’s book is a welcome addition to the literature. It provides both a readable introduction to the fundamental concepts in reinforcement learning as well as intuitive explanations and code for many of the major algorithms in the field. I imagine this will become an invaluable resource for individuals interested in learning about deep reinforcement learning for years to come.”
–Arthur Juliani, senior machine learning engineer, Unity Technologies

“Until now, the only way to get to grips with deep reinforcement learning was to slowly accumulate knowledge from dozens of different sources. Finally, we have a book bringing everything together in one place.”
–Matthew Rahtz, ML researcher, ETH Zürich

Thank you for reading How to Learn Machine Learning, cheers!