© 2018 - Analytics Link

Controlling your keyboard and mouse in Python using pyautogui

June 6, 2018

While recently looking to program an AI to play a (very simple) video game, I needed a way to control the game automatically and to log any keystrokes used so I could train a convolutional neural network to predict which action to take based on what it could see on screen.

 

 

 

 

I came across pyautogui, a Python package which can simulate moving the mouse, clicking the mouse, dragging with the mouse, pressing keys, pressing and holding keys, and pressing keyboard hotkey combinations.

 

I’ve included a couple of examples which showcase some interesting use-cases.  Take a look at the homepage for this package here for a more comprehensive overview of what can be achieved with this package.

 

Example 1 – Drawing

 

Open MS Paint (or another drawing program) open and ensure the drawing space is taking up most of the screen

 

Run the below code in Python

 

Important: You will have 3 seconds to switch the window from Python to Paint.  Once in Paint - hover your cursor where you want the drawing to start

 

import pyautogui

import time

 

#code will sleep for 3 seconds so we can switch over to MS Paint

time.sleep(3)

distance = 200

while distance > 0:

        pyautogui.dragRel(distance, 0, duration=0.2)   # move right

        distance -= 5

        pyautogui.dragRel(0, distance, duration=0.2)   # move down

        pyautogui.dragRel(-distance, 0, duration=0.2)  # move left

        distance -= 5

        pyautogui.dragRel(0, -distance, duration=0.2)  # move up

 

 

 

Example 2 – Opening Notepad and write a message

 

time.sleep(0.5)

pyautogui.hotkey('win')

pyautogui.typewrite(['n','o','t','e','p','a','d','enter'], interval=0.25)

pyautogui.typewrite('Hello World, Pyautogui at your service...', interval=0.25)

 

 

 

This is a pretty noddy way to achieve this task, but it’s in interesting illustration of what pyautogui can do.  As I mentioned above, do take a look here for the complete overview of the package!

 

 

 

 

 

Andrew Jones has over 10 years experience in Analytics, Data Science, and Machine Learning within the Digital, Retail, Telecoms and Travel industries - most recently at Amazon, & Sony PlayStation

Share on Facebook
Please reload

Please reload

Recent Posts