• Andrew Jones

Controlling your keyboard and mouse in Python using pyautogui


While recently looking to program an AI to play a (very simple) video game, I needed a way to control the game automatically and to log any keystrokes used so I could train a convolutional neural network to predict which action to take based on what it could see on screen.

I came across pyautogui, a Python package which can simulate moving the mouse, clicking the mouse, dragging with the mouse, pressing keys, pressing and holding keys, and pressing keyboard hotkey combinations.

I’ve included a couple of examples which showcase some interesting use-cases. Take a look at the homepage for this package here for a more comprehensive overview of what can be achieved with this package.

Example 1 – Drawing

Open MS Paint (or another drawing program) open and ensure the drawing space is taking up most of the screen

Run the below code in Python

Important: You will have 3 seconds to switch the window from Python to Paint. Once in Paint - hover your cursor where you want the drawing to start

import pyautogui

import time

#code will sleep for 3 seconds so we can switch over to MS Paint

time.sleep(3)

distance = 200

while distance > 0:

pyautogui.dragRel(distance, 0, duration=0.2) # move right

distance -= 5

pyautogui.dragRel(0, distance, duration=0.2) # move down

pyautogui.dragRel(-distance, 0, duration=0.2) # move left

distance -= 5

pyautogui.dragRel(0, -distance, duration=0.2) # move up

Example 2 – Opening Notepad and write a message

time.sleep(0.5)

pyautogui.hotkey('win')

pyautogui.typewrite(['n','o','t','e','p','a','d','enter'], interval=0.25)

pyautogui.typewrite('Hello World, Pyautogui at your service...', interval=0.25)

This is a pretty noddy way to achieve this task, but it’s in interesting illustration of what pyautogui can do. As I mentioned above, do take a look here for the complete overview of the package!

Andrew Jones has over 10 years experience in Analytics, Data Science, and Machine Learning within the Digital, Retail, Telecoms and Travel industries - most recently at Amazon, & Sony PlayStation

#python #datascience #andrewjones

© 2018 - Analytics Link