by Al Sweigart
The Command () key (on OS X) 'option' The OPTION key (on OS X)
Pressing and Releasing the Keyboard
Much like the mouseDown() and mouseUp() functions, pyautogui.keyDown() and pyautogui.keyUp() will send virtual keypresses and releases to the computer. They are passed a keyboard key string (see Table 18-1) for their argument. For convenience, PyAutoGUI provides the pyautogui.press() function, which calls both of these functions to simulate a complete keypress.
Run the following code, which will type a dollar sign character (obtained by holding the SHIFT key and pressing 4):
>>> pyautogui.keyDown('shift'); pyautogui.press('4'); pyautogui.keyUp('shift')
This line presses down SHIFT, presses (and releases) 4, and then releases SHIFT. If you need to type a string into a text field, the typewrite() function is more suitable. But for applications that take single-key commands, the press() function is the simpler approach.
Hotkey Combinations
A hotkey or shortcut is a combination of keypresses to invoke some application function. The common hotkey for copying a selection is CTRL-C (on Windows and Linux) or ⌘-C (on OS X). The user presses and holds the CTRL key, then presses the C key, and then releases the C and CTRL keys. To do this with PyAutoGUI’s keyDown() and keyUp() functions, you would have to enter the following:
pyautogui.keyDown('ctrl') pyautogui.keyDown('c') pyautogui.keyUp('c') pyautogui.keyUp('ctrl')
This is rather complicated. Instead, use the pyautogui.hotkey() function, which takes multiple keyboard key string arguments, presses them in order, and releases them in the reverse order. For the CTRL-C example, the code would simply be as follows:
pyautogui.hotkey('ctrl', 'c')
This function is especially useful for larger hotkey combinations. In Word, the CTRL-ALT-SHIFT-S hotkey combination displays the Style pane. Instead of making eight different function calls (four keyDown() calls and four keyUp() calls), you can just call hotkey('ctrl', 'alt', 'shift', 's').
With a new IDLE file editor window in the upper-left corner of your screen, enter the following into the interactive shell (in OS X, replace 'alt' with 'ctrl'):
>>> import pyautogui, time >>> def commentAfterDelay(): ➊ pyautogui.click(100, 100) ➋ pyautogui.typewrite('In IDLE, Alt-3 comments out a line.') time.sleep(2) ➌ pyautogui.hotkey('alt', '3') >>> commentAfterDelay()
This defines a function commentAfterDelay() that, when called, will click the file editor window to bring it into focus ➊, type In IDLE, Atl-3 comments out a line ➋, pause for 2 seconds, and then simulate pressing the ALT-3 hotkey (or CTRL-3 on OS X) ➌. This keyboard shortcut adds two # characters to the current line, commenting it out. (This is a useful trick to know when writing your own code in IDLE.)
Review of the PyAutoGUI Functions
Since this chapter covered many different functions, here is a quick summary reference:
moveTo(x, y). Moves the mouse cursor to the given x and y coordinates.
moveRel(xOffset, yOffset). Moves the mouse cursor relative to its current position.
dragTo(x, y). Moves the mouse cursor while the left button is held down.
dragRel(xOffset, yOffset). Moves the mouse cursor relative to its current position while the left button is held down.
click(x, y, button). Simulates a click (left button by default).
rightClick(). Simulates a right-button click.
middleClick(). Simulates a middle-button click.
doubleClick(). Simulates a double left-button click.
mouseDown(x, y, button). Simulates pressing down the given button at the position x, y.
mouseUp(x, y, button). Simulates releasing the given button at the position x, y.
scroll(units). Simulates the scroll wheel. A positive argument scrolls up; a negative argument scrolls down.
typewrite(message). Types the characters in the given message string.
typewrite([key1, key2, key3]). Types the given keyboard key strings.
press(key). Presses the given keyboard key string.
keyDown(key). Simulates pressing down the given keyboard key.
keyUp(key). Simulates releasing the given keyboard key.
hotkey([key1, key2, key3]). Simulates pressing the given keyboard key strings down in order and then releasing them in reverse order.
screenshot(). Returns a screenshot as an Image object. (See Chapter 17 for information on Image objects.)
Project: Automatic Form Filler
Of all the boring tasks, filling out forms is the most dreaded of chores. It’s only fitting that now, in the final chapter project, you will slay it. Say you have a huge amount of data in a spreadsheet, and you have to tediously retype it into some other application’s form interface—with no intern to do it for you. Although some applications will have an Import feature that will allow you to upload a spreadsheet with the information, sometimes it seems that there is no other way than mindlessly clicking and typing for hours on end. You’ve come this far in this book; you know that of course there’s another way.
The form for this project is a Google Docs form that you can find at http://nostarch.com/automatestuff. It looks like Figure 18-4.
Figure 18-4. The form used for this project
At a high level, here’s what your program should do:
Click the first text field of the form.
Move through the form, typing information into each field.
Click the Submit button.
Repeat the process with the next set of data.
This means your code will need to do the following:
Call pyautogui.click() to click the form and Submit button.
Call pyautogui.typewrite() to enter text into the fields.
Handle the KeyboardInterrupt exception so the user can press CTRL-C to quit.
Open a new file editor window and save it as formFiller.py.
Step 1: Figure Out the Steps
Before writing code, you need to figure out the exact keystrokes and mouse clicks that will fill out the form once. The mouseNow.py script in Project: “Where Is the Mouse Right Now?” can help you figure out specific mouse coordinates. You need to know only the coordinates of the first text field. After clicking the first field, you can just press TAB to move focus to the next field. This will save you from having to figure out the x- and y-coordinates to click for every field.
Here are the steps for entering data into the form:
Click the Name field. (Use mouseNow.py to determine the coordinates after maximizing the browser window. On OS X, you may need to click twice: once to put the browser in focus and again to click the Name field.)
Type a name and then press TAB.
Type a greatest fear and then press TAB.
Press the down arrow key the correct number of times to select the wizard power source: once for wand, twice for amulet, three times for crystal ball, and four times for money. Then press TAB. (Note that on OS X, you will have to press the down arrow key one more time for each option. For some browsers, you may need to press the ENTER key as well.)
Press the right arrow key to select the answer to the RoboCop question. Press it once for 2, twice for 3, three times for 4, or four times for 5; or just press the spacebar to select 1 (which is highlighted by default). Then press TAB.
Type an additional comment and then press TAB.
Press the ENTER key to “click” the Submit button.
After submitting the form, the browser will take you to a page where you will need to click a link to return to the form page.
Note that if you run this program again later, you may have to update the mouse click coordinates, since the browser window might have changed position. To work around this, always make sure the browser window is maximized before finding the coordinates of the first form field. Also, different browsers on different operating systems might work slightly differently from the steps given here, so check that these keystroke combinations work for your computer before running your program.
Step 2: Set
Up Coordinates
Load the example form you downloaded (Figure 18-4) in a browser and maximize your browser window. Open a new Terminal or command line window to run the mouseNow.py script, and then mouse over the Name field to figure out its the x- and y-coordinates. These numbers will be assigned to the nameField variable in your program. Also, find out the x- and y-coordinates and RGB tuple value of the blue Submit button. These values will be assigned to the submitButton and submitButtonColor variables, respectively.
Next, fill in some dummy data for the form and click Submit. You need to see what the next page looks like so that you can use mouseNow.py to find the coordinates of the Submit another response link on this new page.
Make your source code look like the following, being sure to replace all the values in italics with the coordinates you determined from your own tests:
#! python3 # formFiller.py - Automatically fills in the form. import pyautogui, time # Set these to the correct coordinates for your computer. nameField = (648, 319) submitButton = (651, 817) submitButtonColor = (75, 141, 249) submitAnotherLink = (760, 224) # TODO: Give the user a chance to kill the script. # TODO: Wait until the form page has loaded. # TODO: Fill out the Name Field. # TODO: Fill out the Greatest Fear(s) field. # TODO: Fill out the Source of Wizard Powers field. # TODO: Fill out the RoboCop field. # TODO: Fill out the Additional Comments field. # TODO: Click Submit. # TODO: Wait until form page has loaded. # TODO: Click the Submit another response link.
Now you need the data you actually want to enter into this form. In the real world, this data might come from a spreadsheet, a plaintext file, or a website, and it would require additional code to load into the program. But for this project, you’ll just hardcode all this data in a variable. Add the following to your program:
#! python3 # formFiller.py - Automatically fills in the form. --snip-- formData = [{'name': 'Alice', 'fear': 'eavesdroppers', 'source': 'wand', 'robocop': 4, 'comments': 'Tell Bob I said hi.'}, {'name': 'Bob', 'fear': 'bees', 'source': 'amulet', 'robocop': 4, 'comments': 'n/a'}, {'name': 'Carol', 'fear': 'puppets', 'source': 'crystal ball', 'robocop': 1, 'comments': 'Please take the puppets out of the break room.'}, {'name': 'Alex Murphy', 'fear': 'ED-209', 'source': 'money', 'robocop': 5, 'comments': 'Protect the innocent. Serve the public trust. Uphold the law.'}, ] --snip--
The formData list contains four dictionaries for four different names. Each dictionary has names of text fields as keys and responses as values. The last bit of setup is to set PyAutoGUI’s PAUSE variable to wait half a second after each function call. Add the following to your program after the formData assignment statement:
pyautogui.PAUSE = 0.5
Step 3: Start Typing Data
A for loop will iterate over each of the dictionaries in the formData list, passing the values in the dictionary to the PyAutoGUI functions that will virtually type in the text fields.
Add the following code to your program:
#! python3 # formFiller.py - Automatically fills in the form. --snip-- for person in formData: # Give the user a chance to kill the script. print('>>> 5 SECOND PAUSE TO LET USER PRESS CTRL-C <<<') ➊ time.sleep(5) # Wait until the form page has loaded. ➋ while not pyautogui.pixelMatchesColor(submitButton[0], submitButton[1], submitButtonColor): time.sleep(0.5) --snip--
As a small safety feature, the script has a five-second pause ➊ that gives the user a chance to hit CTRL-C (or move the mouse cursor to the upper-left corner of the screen to raise the FailSafeException exception) to shut the program down in case it’s doing something unexpected. Then the program waits until the Submit button’s color is visible ➋, letting the program know that the form page has loaded. Remember that you figured out the coordinate and color information in step 2 and stored it in the submitButton and submitButtonColor variables. To use pixelMatchesColor(), you pass the coordinates submitButton[0] and submitButton[1], and the color submitButtonColor.
After the code that waits until the Submit button’s color is visible, add the following:
#! python3 # formFiller.py - Automatically fills in the form. --snip-- ➊ print('Entering %s info...' % (person['name'])) ➋ pyautogui.click(nameField[0], nameField[1]) # Fill out the Name field. ➌ pyautogui.typewrite(person['name'] + 't') # Fill out the Greatest Fear(s) field. ➍ pyautogui.typewrite(person['fear'] + 't') --snip--
We add an occasional print() call to display the program’s status in its Terminal window to let the user know what’s going on ➊.
Since the program knows that the form is loaded, it’s time to call click() to click the Name field ➋ and typewrite() to enter the string in person['name'] ➌. The 't' character is added to the end of the string passed to typewrite() to simulate pressing TAB, which moves the keyboard focus to the next field, Greatest Fear(s). Another call to typewrite() will type the string in person['fear'] into this field and then tab to the next field in the form ➍.
Step 4: Handle Select Lists and Radio Buttons
The drop-down menu for the “wizard powers” question and the radio buttons for the RoboCop field are trickier to handle than the text fields. To click these options with the mouse, you would have to figure out the x- and y-coordinates of each possible option. It’s easier to use the keyboard arrow keys to make a selection instead.
Add the following to your program:
#! python3 # formFiller.py - Automatically fills in the form. --snip-- # Fill out the Source of Wizard Powers field. ➊ if person['source'] == 'wand': ➋ pyautogui.typewrite(['down', 't']) elif person['source'] == 'amulet': pyautogui.typewrite(['down', 'down', 't']) elif person['source'] == 'crystal ball': pyautogui.typewrite(['down', 'down', 'down', 't']) elif person['source'] == 'money': pyautogui.typewrite(['down', 'down', 'down', 'down', 't']) # Fill out the RoboCop field. ➌ if person['robocop'] == 1: ➍ pyautogui.typewrite([' ', 't']) elif person['robocop'] == 2: pyautogui.typewrite(['right', 't']) elif person['robocop'] == 3: pyautogui.typewrite(['right', 'right', 't']) elif person['robocop'] == 4: pyautogui.typewrite(['right', 'right', 'right', 't']) elif person['robocop'] == 5: pyautogui.typewrite(['right', 'right', 'right', 'right', 't']) --snip--
Once the drop-down menu has focus (remember that you wrote code to simulate pressing TAB after filling out the Greatest Fear(s) field), pressing the down arrow key will move to the next item in the selection list. Depending on the value in person['source'], your program should send a number of down arrow keypresses before tabbing to the next field. If the value at the 'source' key in this user’s dictionary is 'wand' ➊, we simulate pressing the down arrow key once (to select Wand) and pressing TAB ➋. If the value at the 'source' key is 'amulet', we simulate pressing the down arrow key twice and pressing TAB, and so on for the other possible answers.
The radio buttons for the RoboCop question can be selected with the right arrow keys—or, if you want to select the first choice ➌, by just pressing the spacebar ➍.
Step 5: Submit the Form and Wait
You can fill out the Additional Comments field with the typewrite() function by passing person['comments'] as an argument. You can type an additional 't' to move the keyboard focus to the next field or the Submit button. Once the Submit button is in focus, calling pyautogui.press('enter') will simulate pressing the ENTER key and submit the form. After submitting the form, your program will wait five seconds for the next page to load.
Once the new page has loaded, it will have a Submit another response link that will direct the browser to a new, empty form page. You stored the coordinates of this link as a tuple in submitAnotherLink in step 2, so pass these coordinates to pyautogui.click() to click this link.
With the new form ready to go, the script’s outer for loop can continue to the next iteration and enter the next person’s information into the form.
Complete your program by adding the following code:
#! python3 # formFiller.py - Automatically fills in the form. --snip-- # Fill out the Additional Comments field. pyautogui.typewrite(person['comments'] + 't') # Click Submit. pyautogui.
press('enter') # Wait until form page has loaded. print('Clicked Submit.') time.sleep(5) # Click the Submit another response link. pyautogui.click(submitAnotherLink[0], submitAnotherLink[1])
Once the main for loop has finished, the program will have plugged in the information for each person. In this example, there are only four people to enter. But if you had 4,000 people, then writing a program to do this would save you a lot of time and typing!
Summary
GUI automation with the pyautogui module allows you to interact with applications on your computer by controlling the mouse and keyboard. While this approach is flexible enough to do anything that a human user can do, the downside is that these programs are fairly blind to what they are clicking or typing. When writing GUI automation programs, try to ensure that they will crash quickly if they’re given bad instructions. Crashing is annoying, but it’s much better than the program continuing in error.
You can move the mouse cursor around the screen and simulate mouse clicks, keystrokes, and keyboard shortcuts with PyAutoGUI. The pyautogui module can also check the colors on the screen, which can provide your GUI automation program with enough of an idea of the screen contents to know whether it has gotten offtrack. You can even give PyAutoGUI a screen-shot and let it figure out the coordinates of the area you want to click.