While recently looking to program an AI to play a (very simple) video game, I needed a way to control the game automatically and to log any keystrokes used so I could train a convolutional neural network to predict which action to take based on what it could see on screen.
I came across pyautogui, a Python package which can simulate moving the mouse, clicking the mouse, dragging with the mouse, pressing keys, pressing and holding keys, and pressing keyboard hotkey combinations.
I’ve included a couple of examples which showcase some interesting use-cases. Take a look at the homepage for this package here for a more comprehensive overview of what can be achieved with this package.
Example 1 – Drawing
Open MS Paint (or another drawing program) open and ensure the drawing space is taking up most of the screen
Run the below code in Python
Important: You will have 3 seconds to switch the window from Python to Paint. Once in Paint - hover your cursor where you want the drawing to start
import pyautogui
import time
#code will sleep for 3 seconds so we can switch over to MS Paint
time.sleep(3)
distance = 200
while distance > 0:
pyautogui.dragRel(distance, 0, duration=0.2) # move right
distance -= 5
pyautogui.dragRel(0, distance, duration=0.2) # move down
pyautogui.dragRel(-distance, 0, duration=0.2) # move left
distance -= 5
pyautogui.dragRel(0, -distance, duration=0.2) # move up
Example 2 – Opening Notepad and write a message
time.sleep(0.5)
pyautogui.hotkey('win')
pyautogui.typewrite(['n','o','t','e','p','a','d','enter'], interval=0.25)
pyautogui.typewrite('Hello World, Pyautogui at your service...', interval=0.25)
This is a pretty noddy way to achieve this task, but it’s in interesting illustration of what pyautogui can do. As I mentioned above, do take a look here for the complete overview of the package!
Andrew Jones has over 10 years experience in Analytics, Data Science, and Machine Learning within the Digital, Retail, Telecoms and Travel industries - most recently at Amazon, & Sony PlayStation