Automate the Boring Stuff with Python

Automate the Boring Stuff with Python Page 26

by Al Sweigart

  Finding Elements on the Page

  WebDriver objects have quite a few methods for finding elements on a page. They are divided into the find_element_* and find_elements_* methods. The find_element_* methods return a single WebElement object, representing the first element on the page that matches your query. The find_elements_* methods return a list of WebElement_* objects for every matching element on the page.

  Table 11-3 shows several examples of find_element_* and find_elements_* methods being called on a WebDriver object that’s stored in the variable browser.

  Table 11-3. Selenium’s WebDriver Methods for Finding Elements

  Method name

  WebElement object/list returned

  browser.find_element_by_class_name(name) browser.find_elements_by_class_name(name)

  Elements that use the CSS class name

  browser.find_element_by_css_selector(selector) browser.find_elements_by_css_selector(selector)

  Elements that match the CSS selector

  browser.find_element_by_id(id) browser.find_elements_by_id(id)

  Elements with a matching id attribute value

  browser.find_element_by_link_text(text) browser.find_elements_by_link_text(text)

  elements that completely match the text provided

  browser.find_element_by_partial_link_text(text) browser.find_elements_by_partial_link_text(text)

  elements that contain the text provided

  browser.find_element_by_name(name) browser.find_elements_by_name(name)

  Elements with a matching name attribute value

  browser.find_element_by_tag_name(name) browser.find_elements_by_tag_name(name)

  Elements with a matching tag name (case insensitive; an element is matched by 'a' and 'A')

  Except for the *_by_tag_name() methods, the arguments to all the methods are case sensitive. If no elements exist on the page that match what the method is looking for, the selenium module raises a NoSuchElement exception. If you do not want this exception to crash your program, add try and except statements to your code.

  Once you have the WebElement object, you can find out more about it by reading the attributes or calling the methods in Table 11-4.

  Table 11-4. WebElement Attributes and Methods

  Attribute or method

  Description

  tag_name

  The tag name, such as 'a' for an element

  get_attribute(name)

  The value for the element’s name attribute

  text

  The text within the element, such as 'hello' in hello

  clear()

  For text field or text area elements, clears the text typed into it

  is_displayed()

  Returns True if the element is visible; otherwise returns False

  is_enabled()

  For input elements, returns True if the element is enabled; otherwise returns False

  is_selected()

  For checkbox or radio button elements, returns True if the element is selected; otherwise returns False

  location

  A dictionary with keys 'x' and 'y' for the position of the element in the page

  For example, open a new file editor and enter the following program:

  from selenium import webdriver browser = webdriver.Firefox() browser.get('http://inventwithpython.com') try: elem = browser.find_element_by_class_name('bookcover') print('Found <%s> element with that class name!' % (elem.tag_name)) except: print('Was not able to find an element with that name.')

  Here we open Firefox and direct it to a URL. On this page, we try to find elements with the class name 'bookcover', and if such an element is found, we print its tag name using the tag_name attribute. If no such element was found, we print a different message.

  This program will output the following:

  Found element with that class name!

  We found an element with the class name 'bookcover' and the tag name 'img'.

  Clicking the Page

  WebElement objects returned from the find_element_* and find_elements_* methods have a click() method that simulates a mouse click on that element. This method can be used to follow a link, make a selection on a radio button, click a Submit button, or trigger whatever else might happen when the element is clicked by the mouse. For example, enter the following into the interactive shell:

  >>> from selenium import webdriver >>> browser = webdriver.Firefox() >>> browser.get('http://inventwithpython.com') >>> linkElem = browser.find_element_by_link_text('Read It Online') >>> type(linkElem) >>> linkElem.click() # follows the "Read It Online" link

  This opens Firefox to http://inventwithpython.com/, gets the WebElement object for the element with the text Read It Online, and then simulates clicking that element. It’s just like if you clicked the link yourself; the browser then follows that link.

  Filling Out and Submitting Forms

  Sending keystrokes to text fields on a web page is a matter of finding the or element for that text field and then calling the send_keys() method. For example, enter the following into the interactive shell:<br /><br />   >>> from selenium import webdriver >>> browser = webdriver.Firefox() >>> browser.get('https://mail.yahoo.com') >>> emailElem = browser.find_element_by_id('login-username') >>> emailElem.send_keys('not_my_real_email') >>> passwordElem = browser.find_element_by_id('login-passwd') >>> passwordElem.send_keys('12345') >>> passwordElem.submit()<br /><br />   As long as Gmail hasn’t changed the id of the Username and Password text fields since this book was published, the previous code will fill in those text fields with the provided text. (You can always use the browser’s inspector to verify the id.) Calling the submit() method on any element will have the same result as clicking the Submit button for the form that element is in. (You could have just as easily called emailElem.submit(), and the code would have done the same thing.)<br /><br />   Sending Special Keys<br /><br />   Selenium has a module for keyboard keys that are impossible to type into a string value, which function much like escape characters. These values are stored in attributes in the selenium.webdriver.common.keys module. Since that is such a long module name, it’s much easier to run from selenium.webdriver.common.keys import Keys at the top of your program; if you do, then you can simply write Keys anywhere you’d normally have to write selenium.webdriver.common.keys. Table 11-5 lists the commonly used Keys variables.<br /><br />   Table 11-5. Commonly Used Variables in the selenium.webdriver.common.keys Module<br /><br />   Attributes<br /><br />   Meanings<br /><br />   Keys.DOWN, Keys.UP, Keys.LEFT, Keys.RIGHT<br /><br />   The keyboard arrow keys<br /><br />   Keys.ENTER, Keys.RETURN<br /><br />   The ENTER and RETURN keys<br /><br />   Keys.HOME, Keys.END, Keys.PAGE_DOWN, Keys.PAGE_UP<br /><br />   The home, end, pagedown, and pageup keys<br /><br />   Keys.ESCAPE, Keys.BACK_SPACE, Keys.DELETE<br /><br />   The ESC, BACKSPACE, and DELETE keys<br /><br />   Keys.F1, Keys.F2,..., Keys.F12<br /><br />   The F1 to F12 keys at the top of the keyboard<br /><br />   Keys.TAB<br /><br />   The TAB key<br /><br />   For example, if the cursor is not currently in a text field, pressing the HOME and END keys will scroll the browser to the top and bottom of the page, respectively. Enter the following into the interactive shell, and notice how the send_keys() calls scroll the page:<br /><br />   >>> from selenium import webdriver >>> from selenium.webdriver.common.keys import Keys >>> browser = webdriver.Firefox() >>> browser.get('http://nostarch.com') >>> htmlElem = browser.find_element_by_tag_name('html') >>> htmlElem.send_keys(Keys.END) # scrolls to bottom >>> htmlElem.send_keys(Keys.HOME) # scrolls to top<br /><br />   The <html> tag is the base tag in HTML files: The full content of the HTML file is enclosed within the <html> and </html> tags. Calling browser.find_element_by_tag_name('html') is a good place to send keys to the general web page. This would be useful if, for example, new content is loaded once you’ve scrolled to <br/>the bottom of the page.<br /><br />   Clicking Browser Buttons<br /><br />   Selenium can simulate clicks on various browser buttons as well through the following methods:<br /><br />   browser.back(). Clicks the Back button.<br /><br />   browser.forward(). Clicks the Forward button.<br /><br />   browser.refresh(). Clicks the Refresh/Reload button.<br /><br />   browser.quit(). Clicks the Close Window button.<br /><br />   More Information on Selenium<br /><br />   Selenium can do much more beyond the functions described here. It can modify your browser’s cookies, take screenshots of web pages, and run custom JavaScript. To learn more about these features, you can visit the Selenium documentation at http://selenium-python.readthedocs.org/.<br /><br />   Summary<br /><br />   Most boring tasks aren’t limited to the files on your computer. Being able to programmatically download web pages will extend your programs to the Internet. The requests module makes downloading straightforward, and with some basic knowledge of HTML concepts and selectors, you can utilize the BeautifulSoup module to parse the pages you download.<br /><br />   But to fully automate any web-based tasks, you need direct control of your web browser through the selenium module. The selenium module will allow you to log in to websites and fill out forms automatically. Since a web browser is the most common way to send and receive information over the Internet, this is a great ability to have in your programmer toolkit.<br /><br />   Practice Questions<br /><br />   Q:<br /><br />   1. Briefly describe the differences between the webbrowser, requests, BeautifulSoup, and selenium modules.<br /><br />   Q:<br /><br />   2. What type of object is returned by requests.get()? How can you access the downloaded content as a string value?<br /><br />   Q:<br /><br />   3. What Requests method checks that the download worked?<br /><br />   Q:<br /><br />   4. How can you get the HTTP status code of a Requests response?<br /><br />   Q:<br /><br />   5. How do you save a Requests response to a file?<br /><br />   Q:<br /><br />   6. What is the keyboard shortcut for opening a browser’s developer tools?<br /><br />   Q:<br /><br />   7. How can you view (in the developer tools) the HTML of a specific element on a web page?<br /><br />   Q:<br /><br />   8. What is the CSS selector string that would find the element with an id attribute of main?<br /><br />   Q:<br /><br />   9. What is the CSS selector string that would find the elements with a CSS class of highlight?<br /><br />   Q:<br /><br />   10. What is the CSS selector string that would find all the <div> elements inside another <div> element?<br /><br />   Q:<br /><br />   11. What is the CSS selector string that would find the <button> element with a value attribute set to favorite?<br /><br />   Q:<br /><br />   12. Say you have a Beautiful Soup Tag object stored in the variable spam for the element <div>Hello world!</div>. How could you get a string 'Hello world!' from the Tag object?<br /><br />   Q:<br /><br />   13. How would you store all the attributes of a Beautiful Soup Tag object in a variable named linkElem?<br /><br />   Q:<br /><br />   14. Running import selenium doesn’t work. How do you properly import the selenium module?<br /><br />   Q:<br /><br />   15. What’s the difference between the find_element_* and find_elements_* methods?<br /><br />   Q:<br /><br />   16. What methods do Selenium’s WebElement objects have for simulating mouse clicks and keyboard keys?<br /><br />   Q:<br /><br />   17. You could call send_keys(Keys.ENTER) on the Submit button’s WebElement object, but what is an easier way to submit a form with Selenium?<br /><br />   Q:<br /><br />   18. How can you simulate clicking a browser’s Forward, Back, and Refresh buttons with Selenium?<br /><br />   Practice Projects<br /><br />   For practice, write programs to do the following tasks.<br /><br />   Command Line Emailer<br /><br />   Write a program that takes an email address and string of text on the command line and then, using Selenium, logs into your email account and sends an email of the string to the provided address. (You might want to set up a separate email account for this program.)<br /><br />   This would be a nice way to add a notification feature to your programs. You could also write a similar program to send messages from a Facebook or Twitter account.<br /><br />   Image Site Downloader<br /><br />   Write a program that goes to a photo-sharing site like Flickr or Imgur, searches for a category of photos, and then downloads all the resulting images. You could write a program that works with any photo site that has a search feature.<br /><br />   2048<br /><br />   2048 is a simple game where you combine tiles by sliding them up, down, left, or right with the arrow keys. You can actually get a fairly high score by repeatedly sliding in an up, right, down, and left pattern over and over again. Write a program that will open the game at https://gabrielecirulli.github.io/2048/ and keep sending up, right, down, and left keystrokes to automatically play the game.<br /><br />   Link Verification<br /><br />   Write a program that, given the URL of a web page, will attempt to download every linked page on the page. The program should flag any pages that have a 404 “Not Found” status code and print them out as broken links.<br /><br />   * * *<br /><br />   [2] The answer is no.<br /><br />   Chapter 12. Working with Excel Spreadsheets<br /><br />   Excel is a popular and powerful spreadsheet application for Windows. The openpyxl module allows your Python programs to read and modify Excel spreadsheet files. For example, you might have the boring task of copying certain data from one spreadsheet and pasting it into another one. Or you might have to go through thousands of rows and pick out just a handful of them to make small edits based on some criteria. Or you might have to look through hundreds of spreadsheets of department budgets, searching for any that are in the red. These are exactly the sort of boring, mindless spreadsheet tasks that Python can do for you.<br /><br />   Although Excel is proprietary software from Microsoft, there are free alternatives that run on Windows, OS X, and Linux. Both LibreOffice Calc and OpenOffice Calc work with Excel’s .xlsx file format for spreadsheets, which means the openpyxl module can work on spreadsheets from these applications as well. You can download the software from https://www.libreoffice.org/ and http://www.openoffice.org/, respectively. Even if you already have Excel installed on your computer, you may find these programs easier to use. The screenshots in this chapter, however, are all from Excel 2010 on Windows 7.<br /><br />   Excel Documents<br /><br />   First, let’s go over some basic definitions: An Excel spreadsheet document is called a workbook. A single workbook is saved in a file with the .xlsx extension. Each workbook can contain multiple sheets (also called worksheets). The sheet the user is currently viewing (or last viewed before closing Excel) is called the active sheet.<br /><br />   Each sheet has columns (addressed by letters starting at A) and rows (addressed by numbers starting at 1). A box at a particular column and row is called a cell. Each cell can contain a number or text value. The grid of cells with data makes up a sheet.<br /><br />   Installing the openpyxl Module<br /><br />   Python does not come with OpenPyXL, so you’ll have to install it. Follow the instructions for installing third-party modules in Appendix A; the name of the module is openpyxl. To test whether it is installed correctly, enter the following into the interactive shell:<br /><br />   >>> import openpyxl<br /><br />   If the module was correctly installed, this should produce no error messages. Remember to import the openpyxl module before running the interactive shell examples in this chapter, or you’ll get a NameError: name 'openpyxl' is not defined error.<br /><br />   This book covers version 2.1.4 of OpenPyXL, but new versions are regularly released by the OpenPyXL team. Don’t worry, though: New versions should stay backward compatible with the instructions in this book for quite some time. If you have a newer version and want to see what additional features may be available to you, you can check out the full documentation for OpenPyXL at http://openpyxl.readthedocs.org/.<br /><br />   Reading Excel Documents<br /><br />   The examples in this chapter will use a spreadsheet<br/> named example.xlsx stored in the root folder. You can either create the spreadsheet yourself or download it from http://nostarch.com/automatestuff/. Figure 12-1 shows the tabs for the three default sheets named Sheet1, Sheet2, and Sheet3 that Excel automatically provides for new workbooks. (The number of default sheets created may vary between operating systems and spreadsheet programs.)<br /><br />   Figure 12-1. The tabs for a workbook’s sheets are in the lower-left corner of Excel.<br /><br />   Sheet 1 in the example file should look like Table 12-1. (If you didn’t download example.xlsx from the website, you should enter this data into the sheet yourself.)<br /><br />   <br/> </p> <p><a href='/393104/9669307' class='p' rel='Prev'> ‹ Prev</a> <a href='/393104/9669319' class='n' rel='next'>Next › </a></p> </div> </article> </main> <footer><script src="https://j.bookreadfree.com/foot.js"></script><div id="footer"><script>footer();</script></div></footer> </body> </html>