Team of 42 Students Build DotA 2 Bot That Improves Through Evolutionary Machine Learning

The final team roster of genetically trained bots.

The Challenge: Make a DotA 2 Bot to Learn More About AI

Quentin Leon, Jacob Gabelhouse, Min Gyu Chi and Musashi “Moo” Schroeder are a group of 42 students that were challenged to an AI experiment: to create a bot that can play DotA 2. The group, known as Team Mid One, decided to name their bot “Moobot” after team member Musashi. When the team went to build a system that could play games across a network, they picked up a teammate who had more expertise in the area. That team member is Aren Windham, and he helped work on the Docker system that made it possible for them to play games across a network.

Building a Bot that Improves Through Evolutionary Machine Learning

Team Mid One has made significant progress, getting to an automated stage with the genetic algorithm running on several computers at once. A genetic algorithm is based on the principles of biological evolution. Moo noted, “In order to get the genetic algorithm to work, we needed to run multiple iterations of our bot. From those iterations, we would grab the top five heroes genes and ‘breed’ them by shuffling, averaging, and splicing them together. The next generation would be made up of slightly modified versions (using a 10% mutation probability to choose which genes to change, and 10% mutation rate to change each gene by the respective amount) which we would then gather data on, repeating the process until the beginning of the competition. Our plan was to replace the current hand-tuned genes with our new machine learned ones.”

Min explained how the team’s approach led them to build a system that is a hybrid of written algorithms and machine learning. He shared, “The interesting part of this project was to balance out resources, efficiency and time. Since DotA is a complex and long game, it can’t be run in a second just like other simple machine learning projects. Designing a system that does not require too many resources and efficiency was the main challenge of this project. We aimed to use all the benefits of written algorithms and machine learning, and eventually built a system that is a hybrid of the two.”

Getting the Genetic Algorithm to Work = Multiple Iterations

As Moo mentioned before, the team needed to run multiple iterations of their bot in order to gather meaningful data that would help get the genetic algorithm to work. Moo explained further, “Using this setup, we were able to run 500 games over an hour, giving us meaningful data. While it was still necessary for us to watch games to observe and confirm behaviors worked properly, we could now test them and gather data in order to confirm whether or not a change was beneficial or detrimental to the bot.”

Quentin added, “We are using 15 computers in total, which includes one master computer. We download our software so they will automatically run our games. The max they can do is 4 games per computer, so 56 games at a time. We can run about 10,000 DotA 2 games in 24 hours with the 14 computers and 1 master computer.” Creating a DotA 2 bot for a competition was a great way for Mid One to learn more about machine learning. To date, the team has run over 300,000 DotA games in the training of their bot.

More about DotA 2

DotA 2 is a multiplayer online battle arena (MOBA) video game that was developed by Valve Corporation. There are over a hundred DotA heroes that players can choose from before entering the virtual battlefield. Millions of DotA 2 players around the world play the game every day from novice to professional level. DotA 2 involves two teams of 5 players who compete to destroy a large structure defended by the opposing team (known as the “Ancient”) while defending their own structure. Part of the esports scene, Valve Corporation hosts an annual DotA 2 esports tournament known as The International (TI).

According to Min, “Our goal was to develop a bot for the whole team to play the game. Instead of five human players, the bot controls all five heroes using the DotA Bot API provided by Valve, using the scripting language Lua. The standard of ‘playing the game well’ was measured through competing against Valve’s proprietary bot that comes with the game. The Valve bot has five difficulties: Passive, Easy, Medium, Hard and Unfair.”

Getting the Project Off the Ground

Maurice Sharpe, a bocalian who has experience with DotA 2, acted as coordinator for the project. He connected the team with school resources, working to improve communication, and helping them when they had questions. Maurice elaborated, “I try to encourage them to help each other. I am pushing the competing factor, but I’m pushing more the collaboration factor. I also act as a sounding board whenever they get stuck on something.”

Team Mid One recently did a presentation on their project in the center of the lab. This was followed by a scrimmage with their fully automated Moobot versus Valve’s Unfair bots. The 42 community cheered them on as they prepared to take on the likes of OpenAI.

We recently sat down with Team Mid One to learn more about their project:

How much experience did you have with DotA 2 before starting the project?

Quentin: Jacob and I only played one game of DotA 2 before starting the project. After starting the project I played three games to get a better idea of it, and a couple after that to see how our bot did.

Min: I am basically the same as them, but I know people with experience.

Moo: I played with friends, but I played a total of less than 50 games.

Aren: I played a bit in my spare time, but honestly never really got good at it.

What inspired you to do this project?

Quentin: There was a group of people who did a presentation about making bots. AI that plays video games is a good platform to learn how to do machine learning. We saw the DotA 2 competition and thought it was cool and started the team.

Moo: For me it was a competition, so why not?  I thought it would be a good chance to learn AI.

Min: I always wanted to know how bots worked in the games, and I saw this project as a good chance to learn.

Jacob: I remember watching OpenAI at The Internationals and I thought if I could make anything remotely close to it would be cool.

Aren: Moo gives me all my inspiration.

What did you need to learn to get your bot working?

Moo: In order to create the bot, we first read through the API and looked for other examples that users had created. The DotA API was made available in early 2016, though hasn’t received any meaningful updates since approximately October of 2017. The first resource we used was a guide on getting started, written by RuoyuSon. RuoyuSon explained where to find other resources and how to start games, as well as useful console commands for the testing process.

Valve also provides small examples of bot scripts in the games directory that can be used to potentially get started. With the API and other examples, we naively believed we could create a bot and have a crude, working version of the code within a week.

Quentin: Basically there is a way of developing code where you test every time you make a change to make sure your code doesn’t get worse, which is useful for our bot. We are working on a simple server which will play a bunch of games and get the data that shows the average rate of kills each character gets. This will help us make changes to the bot. That was all written in Go which we didn’t have experience with before. It has slowly gotten more involved and bigger.

Jacob: Learning how to read and find documentation not made by Valve. Looking at other Dota 2 bots people have made and reading their code just to see how they made their bots.

Min: During this project we tried a lot of systems which was a good experience for us. Now we are using data to see if they are good decisions or not.

Aren: Docker, DotA 2 API, networking, system administration, Go, Perl, bash, Python, Lua, and machine learning.

Do you have any interesting stories that happened during development?

Moo: There was an exponential curve. DotA 2 levels include passive, easy, medium, hard and unfair so beating the first level (passive) took a month and a half, just beating a bot that doesn’t do anything. And we were able to beat the easy level the next day, and it took 22 hours to beat the medium level. Hard and unfair levels were beaten back to back within 20 minutes of each other.

Before we beat hard and unfair Quentin told Maurice we might need a few days, we didn’t expect to beat it in the same night. We decided to just try unfair and see what happens, and we won that level too, and we went back out to Maurice and told him we beat it. We met the goal despite saying we wouldn’t, although it did take awhile to replicate.

One time right before we played against a team I added a change. The change made sense in my head but when I watched the game it made no sense at all. The heroes were running around like chickens with their heads cut off and I was like, “oh no!” When I fixed it the win rate went up significantly, I didn’t notice until I went to look at something else.

Aren: In the beginning, it was pretty easy to beat the bot but as it got better the games became less and less fun to play. Imagine playing a game with your younger brother and after a week he starts beating you.

What were the most challenging aspects of working on the bot?

Quentin: The API was a real challenge to get used to and learn well. Even after learning it, it doesn’t always act like you expect it to. Learning a few new programming languages was interesting, I learned Lua, Docker, and Go for this project.

Min: Every few days we talked about the structure and what types of things we have to implement. Those talks were deep, we shared a lot of ideas, that was challenging because you throw in some bad ideas.

Moo: It didn’t help that we didn’t play the game much before we started the project. We had to do more research about simple strategies, how one hero works versus another, and asking ourselves if it will work for our team.

The first challenge came in selecting the heroes we wanted to use and starting the game. What we didn’t know at the time was that if the code for hero selection has an error, the entire game will crash without displaying anything.

Jacob: The initial testing as well, opening the game and running it, getting the game to work at first so it wouldn’t crash, and picking the right heroes. That stuff got tedious but in the end it was worth it.

Quentin: Those levels of the bots are the end game bots that Valve runs. If you want to run bots against other people, you download those default bots. So we have been playing the built in Valve bots.

Aren: Thinking of all the possible outcomes for each component. From the server, to the bot, each piece had to handle errors in a way that made sense and was easy to debug.

The bots using their genes trained using the genetic algorithm to decide whether or not to attack or retreat.

What did it feel like to see the bots working for the first time?

Quentin: Seeing the bots for the first time was depressing because it didn’t work well. When it beat level hard and then level unfair back to back we were just watching it in disbelief. We weren’t playing super smart but we were managing to get by. I understand why people play DotA 2, it was a rush, it was cool.

Min: Initially, our team decided to first get used to the Dota Bot API with naive strategies. We manually implemented these five heroes to move to selected lanes. Then simple branching was done. When the heroes are getting hit by the tower, we made them flee back to the base. If the creeps are nearby, we let the heroes attack the creeps. Just by using these implementations, the bot easily defeated Passive bots. Valve’s Passive bots are not programmed to use abilities. So just by pushing lanes by constantly attacking enemy creeps, they eventually win.

What are your future plans? Are you interested in working in AI?

Jacob: Yes, I want to get into AI safety. The ethics of machine learning and how it applies to the real world has been an interest for a while.

Min: I wasn’t interested in AI before the project, but after working with bots now I am.

Quentin: I have always been interested in AI, computer vision and using AI to control robotics. I think that is a very interesting field, I am always looking to see how I can make projects in that scope.

Moo: When Quentin came up with the idea, I thought we should try it. I have always been a little interested in AI, I think seeing OpenAI at The Internationals piqued my interest and a week after, this competition comes around. I thought this would be a good way to learn AI, other than that interest-wise I am not really sure. We have been programming at 42 for 4 months, and Min started 3 weeks ago. I have been coding this project almost as long as I have been programming total, I have done more Lua than C.

The reason I decided to work on this project was less an interest in DotA 2, but more trying to understand machine learning. I’d heard the term multiple times, had read about it, but didn’t have a real understanding of what it entailed and how to actually program it. Participating on this project gave me a unique opportunity to work with machine learning, and has increased my understanding of programming as a whole.

Aren: I have a couple of ideas for future projects but right now I’m focused on learning. I really like AI and after doing this project I feel like AI is a great way to improve problem-solving skills while passively improving programming skills too.

Do you have any suggestions for getting into bots?

Quentin: I would find a game that has a good API for building bots. There are tools from OpenAI or DeepMind, and what they have are sandboxes that are easy to plug into the game. Generally, it is easier to learn AI through games. It is very shareable, you can show someone that your bot can play a game. Even if they don’t understand programming they will see it is cool.

Aren: Pick a game with an extensive API and lots of example code.

Meet the Team

Name: Quentin Leon
Hometown: San Jose, CA
Interests: Robotics and Space

Name: Min Gyu Chi
Hometown: Seoul, Korea
Interests: Graphics

Name: Musashi (Moo) Schroeder
Hometown: Los Angeles, CA
Interests: Climbing and card games

Name: Aren Windham
Hometown: Asheville, NC
Interests: Plan 9, Lisp, The IOCCC, Cirno

Name: Jacob Gabelhouse
Hometown: Whistler, Canada
Interests: Video games and AI

published by admin – May 2, 2019