Play Game With Computer Vision Save Abandoned
A simple python bot (powered by computer vision) used to play a game (City Island 5). The bot is able to play the game and collect points without any human intervention.
Gaming using Computer Vision
A python bot used to play a game using computer vision and image processing techniques. The bot is able to play the game without any human intervention. It managed to make me a millionaire in the game overnight!
Here's a screen recording of the game play:
Access a comprehensive guide in this article: https://paulonteri.com/thoughts/play-game-with-computer-vision.
Disclaimer - this content is for educational purposes only!
Why?
I've been playing strategy + city building + simulation? games like TownsMen 6, Clash of the Clans, SimCity for the last 10 years.
On trying out City Island 5 I found it mildly irritating that my collectables could not accumulate while I was outside the game. I might have had the best businesses, strategy, etc but I had to be in the game to ensure I collect the cash/keys/gold overtime. For example, if my bakery makes €100 per minute I would only earn €100 after leaving the game and coming back 24 hours later.
This became especially tiresome while trying to accumulate €5,000,000 required to buy the island shown below. This would take me roughly two weeks of gameplay if I don't spend any money - it's not worth it.
How it works
1. Capture the live game feed
I needed a way to capture the live game feed.
The easiest way to to capture an in-game screenshot and pass it to the next steps in the script.
2. Identify the valuables in the screenshot
We need a way to detect a valuable in the game's feed and then return its coordinates.
OpenCv's Template Matching algorithms are perfect for this.
They are used for searching and finding the location of a template image (like a valuable) in a larger image (like the game's feed). It simply slides the template image over the input image (as in 2D convolution) and compares the template and patch of the input image under the template image. Several comparison methods are implemented in OpenCV. (You can check docs for more details). We use it in the method: cv2.matchTemplate(... )
To achieve this, I needed the template images. I took screenshots by hand and then cropped off the cash, star and key:
3. Collect the valuables by clicking on them
Once we have the coordinates of an item we can try to click on it.
pyautogui's .click(x,y) function works like magic for this. It clicks the screen on the coordinates x and y where our valuable is lying.
Learn more about it here.
4. Close any popups that may appear
Our clicks above may result in popups when we are being given a reward, levelling up, etc.
We need to close it before attempting to collect valuables again. We use the same logic used in finding and clicking on valuables.
To achieve this, I needed the template images for the popups' close buttons so that they can be clicked. I took screenshots by hand and then cropped off the various close buttons:
5. Repeat
We do the steps above repeatedly to collect the valuables while the script is running.
Results after running overnight
I started the game with €316,415.
The following morning I had €6,463,870.
I made €6,147,455 overnight!
I then proceeded to buy the Island I wanted:
Regrets
- This is cheating.
- Why get a game if you're not the one playing it?
Contributing
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
Maintainers
Current maintainers:
- Paul Onteri - https://paulonteri.com
License
Conclusion
Access the full guide in this article here: https://paulonteri.com/thoughts/play-game-with-computer-vision
This was fun!
Open Source Agenda Badge
