Automate the Boring Stuff with Python

by Al Sweigart

>> imapObj.expunge() ('Success', [(5452, 'EXISTS')])

  Here we select the inbox by calling select_folder() on the IMAPClient object and passing 'INBOX' as the first argument; we also pass the keyword argument readonly=False so that we can delete emails ➊. We search the inbox for messages received on a specific date and store the returned message IDs in UIDs ➋. Calling delete_message() and passing it UIDs returns a dictionary; each key-value pair is a message ID and a tuple of the message’s flags, which should now include Deleted ➌. Calling expunge() then permanently deletes messages with the Deleted flag and returns a success message if there were no problems expunging the emails. Note that some email providers, such as Gmail, automatically expunge emails deleted with delete_messages() instead of waiting for an expunge command from the IMAP client.

  Disconnecting from the IMAP Server

  When your program has finished retrieving or deleting emails, simply call the IMAPClient’s logout() method to disconnect from the IMAP server.

  >>> imapObj.logout()

  If your program runs for several minutes or more, the IMAP server may time out, or automatically disconnect. In this case, the next method call your program makes on the IMAPClient object will raise an exception like the following:

  imaplib.abort: socket error: [WinError 10054] An existing connection was forcibly closed by the remote host

  In this event, your program will have to call imapclient.IMAPClient() to connect again.

  Whew! That’s it. There were a lot of hoops to jump through, but you now have a way to get your Python programs to log in to an email account and fetch emails. You can always consult the overview in Retrieving and Deleting Emails with IMAP whenever you need to remember all of the steps.

  Project: Sending Member Dues Reminder Emails

  Say you have been “volunteered” to track member dues for the Mandatory Volunteerism Club. This is a truly boring job, involving maintaining a spreadsheet of everyone who has paid each month and emailing reminders to those who haven’t. Instead of going through the spreadsheet yourself and copying and pasting the same email to everyone who is behind on dues, let’s—you guessed it—write a script that does this for you.

  At a high level, here’s what your program will do:

  Read data from an Excel spreadsheet.

  Find all members who have not paid dues for the latest month.

  Find their email addresses and send them personalized reminders.

  This means your code will need to do the following:

  Open and read the cells of an Excel document with the openpyxl module. (See Chapter 12 for working with Excel files.)

  Create a dictionary of members who are behind on their dues.

  Log in to an SMTP server by calling smtplib.SMTP(), ehlo(), starttls(), and login().

  For all members behind on their dues, send a personalized reminder email by calling the sendmail() method.

  Open a new file editor window and save it as sendDuesReminders.py.

  Step 1: Open the Excel File

  Let’s say the Excel spreadsheet you use to track membership dues payments looks like Figure 16-2 and is in a file named duesRecords.xlsx. You can download this file from http://nostarch.com/automatestuff/.

  Figure 16-2. The spreadsheet for tracking member dues payments

  This spreadsheet has every member’s name and email address. Each month has a column tracking members’ payment statuses. The cell for each member is marked with the text paid once they have paid their dues.

  The program will have to open duesRecords.xlsx and figure out the column for the latest month by calling the get_highest_column() method. (You can consult Chapter 12 for more information on accessing cells in Excel spreadsheet files with the openpyxl module.) Enter the following code into the file editor window:

  #! python3 # sendDuesReminders.py - Sends emails based on payment status in spreadsheet. import openpyxl, smtplib, sys # Open the spreadsheet and get the latest dues status. ➊ wb = openpyxl.load_workbook('duesRecords.xlsx') ➋ sheet = wb.get_sheet_by_name('Sheet1') ➌ lastCol = sheet.get_highest_column() ➍ latestMonth = sheet.cell(row=1, column=lastCol).value # TODO: Check each member's payment status. # TODO: Log in to email account. # TODO: Send out reminder emails.

  After importing the openpyxl, smtplib, and sys modules, we open our duesRecords.xlsx file and store the resulting Workbook object in wb ➊. Then we get Sheet 1 and store the resulting Worksheet object in sheet ➋. Now that we have a Worksheet object, we can access rows, columns, and cells. We store the highest column in lastCol ➌, and we then use row number 1 and lastCol to access the cell that should hold the most recent month. We get the value in this cell and store it in latestMonth ➍.

  Step 2: Find All Unpaid Members

  Once you’ve determined the column number of the latest month (stored in lastCol), you can loop through all rows after the first row (which has the column headers) to see which members have the text paid in the cell for that month’s dues. If the member hasn’t paid, you can grab the member’s name and email address from columns 1 and 2, respectively. This information will go into the unpaidMembers dictionary, which will track all members who haven’t paid in the most recent month. Add the following code to sendDuesReminder.py.

  #! python3 # sendDuesReminders.py - Sends emails based on payment status in spreadsheet. --snip-- # Check each member's payment status. unpaidMembers = {} ➊ for r in range(2, sheet.get_highest_row() + 1): ➋ payment = sheet.cell(row=r, column=lastCol).value if payment != 'paid': ➌ name = sheet.cell(row=r, column=1).value ➍ email = sheet.cell(row=r, column=2).value ➎ unpaidMembers[name] = email

  This code sets up an empty dictionary unpaidMembers and then loops through all the rows after the first ➊. For each row, the value in the most recent column is stored in payment ➋. If payment is not equal to 'paid', then the value of the first column is stored in name ➌, the value of the second column is stored in email ➍, and name and email are added to unpaidMembers ➎.

  Step 3: Send Customized Email Reminders

  Once you have a list of all unpaid members, it’s time to send them email reminders. Add the following code to your program, except with your real email address and provider information:

  #! python3 # sendDuesReminders.py - Sends emails based on payment status in spreadsheet. --snip-- # Log in to email account. smtpObj = smtplib.SMTP('smtp.gmail.com', 587) smtpObj.ehlo() smtpObj.starttls() smtpObj.login(' my_email_address@gmail.com ', sys.argv[1])

  Create an SMTP object by calling smtplib.SMTP() and passing it the domain name and port for your provider. Call ehlo() and starttls(), and then call login() and pass it your email address and sys.argv[1], which will store your password string. You’ll enter the password as a command line argument each time you run the program, to avoid saving your password in your source code.

  Once your program has logged in to your email account, it should go through the unpaidMembers dictionary and send a personalized email to each member’s email address. Add the following to sendDuesReminders.py:

  #! python3 # sendDuesReminders.py - Sends emails based on payment status in spreadsheet. --snip-- # Send out reminder emails. for name, email in unpaidMembers.items(): ➊ body = "Subject: %s dues unpaid.nDear %s,nRecords show that you have not paid dues for %s. Please make this payment as soon as possible. Thank you!'" % (latestMonth, name, latestMonth) ➋ print('Sending email to %s...' % email) ➌ sendmailStatus = smtpObj.sendmail('my_email_address@gmail.com', email, body) ➍ if sendmailStatus != {}: print('There was a problem sending email to %s: %s' % (email, sendmailStatus)) smtpObj.quit()

  This code loops through the names and emails in unpaidMembers. For each member who hasn’t paid, we customize a message with the latest month and the member’s name, and store the message in body ➊. We print output saying that we’re sending an email to this member’s email address ➋. Then we call sendmail(), passing it the from address and the customized message ➌. We store the return value in sendmailStatus.

  Remember t
hat the sendmail() method will return a nonempty dictionary value if the SMTP server reported an error sending that particular email. The last part of the for loop at ➍ checks if the returned dictionary is nonempty, and if it is, prints the recipient’s email address and the returned dictionary.

  After the program is done sending all the emails, the quit() method is called to disconnect from the SMTP server.

  When you run the program, the output will look something like this:

  Sending email to alice@example.com... Sending email to bob@example.com... Sending email to eve@example.com...

  The recipients will receive an email that looks like Figure 16-3.

  Figure 16-3. An automatically sent email from sendDuesReminders.py

  Sending Text Messages with Twilio

  Most people are more likely to be near their phones than their computers, so text messages can be a more immediate and reliable way of sending notifications than email. Also, the short length of text messages makes it more likely that a person will get around to reading them.

  In this section, you’ll learn how to sign up for the free Twilio service and use its Python module to send text messages. Twilio is an SMS gateway service, which means it’s a service that allows you to send text messages from your programs. Although you will be limited in how many texts you can send per month and the texts will be prefixed with the words Sent from a Twilio trial account, this trial service is probably adequate for your personal programs. The free trial is indefinite; you won’t have to upgrade to a paid plan later.

  Twilio isn’t the only SMS gateway service. If you prefer not to use Twilio, you can find alternative services by searching online for free sms gateway, python sms api, or even twilio alternatives.

  Before signing up for a Twilio account, install the twilio module. Appendix A has more details about installing third-party modules.

  Note

  This section is specific to the United States. Twilio does offer SMS texting services for countries outside of the United States, but those specifics aren’t covered in this book. The twilio module and its functions, however, will work the same outside the United States. See http://twilio.com/ for more information.

  Signing Up for a Twilio Account

  Go to http://twilio.com/ and fill out the sign-up form. Once you’ve signed up for a new account, you’ll need to verify a mobile phone number that you want to send texts to. (This verification is necessary to prevent people from using the service to spam random phone numbers with text messages.)

  After receiving the text with the verification number, enter it into the Twilio website to prove that you own the mobile phone you are verifying. You will now be able to send texts to this phone number using the twilio module.

  Twilio provides your trial account with a phone number to use as the sender of text messages. You will need two more pieces of information: your account SID and the auth (authentication) token. You can find this information on the Dashboard page when you are logged in to your Twilio account. These values act as your Twilio username and password when logging in from a Python program.

  Sending Text Messages

  Once you’ve installed the twilio module, signed up for a Twilio account, verified your phone number, registered a Twilio phone number, and obtained your account SID and auth token, you will finally be ready to send yourself text messages from your Python scripts.

  Compared to all the registration steps, the actual Python code is fairly simple. With your computer connected to the Internet, enter the following into the interactive shell, replacing the accountSID, authToken, myTwilioNumber, and myCellPhone variable values with your real information:

  ➊ >>> from twilio.rest import TwilioRestClient >>> accountSID = 'ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' >>> authToken = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' ➋ >>> twilioCli = TwilioRestClient(accountSID, authToken) >>> myTwilioNumber = '+14955551234' >>> myCellPhone = '+14955558888' ➌ >>> message = twilioCli.messages.create(body='Mr. Watson - Come here - I want to see you.', from_=myTwilioNumber, to=myCellPhone)

  A few moments after typing the last line, you should receive a text message that reads Sent from your Twilio trial account - Mr. Watson - Come here - I want to see you.

  Because of the way the twilio module is set up, you need to import it using from twilio.rest import TwilioRestClient, not just import twilio ➊. Store your account SID in accountSID and your auth token in authToken and then call TwilioRestClient() and pass it accountSID and authToken. The call to TwilioRestClient() returns a TwilioRestClient object ➋. This object has a messages attribute, which in turn has a create() method you can use to send text messages. This is the method that will instruct Twilio’s servers to send your text message. After storing your Twilio number and cell phone number in myTwilioNumber and myCellPhone, respectively, call create() and pass it keyword arguments specifying the body of the text message, the sender’s number (myTwilioNumber), and the recipient’s number (myCellPhone) ➌.

  The Message object returned from the create() method will have information about the text message that was sent. Continue the interactive shell example by entering the following:

  >>> message.to '+14955558888' >>> message.from_ '+14955551234' >>> message.body 'Mr. Watson - Come here - I want to see you.'

  The to, from_, and body attributes should hold your cell phone number, Twilio number, and message, respectively. Note that the sending phone number is in the from_ attribute—with an underscore at the end—not from. This is because from is a keyword in Python (you’ve seen it used in the from modulename import * form of import statement, for example), so it cannot be used as an attribute name. Continue the interactive shell example with the following:

  >>> message.status 'queued' >>> message.date_created datetime.datetime(2015, 7, 8, 1, 36, 18) >>> message.date_sent == None True

  The status attribute should give you a string. The date_created and date_sent attributes should give you a datetime object if the message has been created and sent. It may seem odd that the status attribute is set to 'queued' and the date_sent attribute is set to None when you’ve already received the text message. This is because you captured the Message object in the message variable before the text was actually sent. You will need to refetch the Message object in order to see its most up-to-date status and date_sent. Every Twilio message has a unique string ID (SID) that can be used to fetch the latest update of the Message object. Continue the interactive shell example by entering the following:

  >>> message.sid 'SM09520de7639ba3af137c6fcb7c5f4b51' ➊ >>> updatedMessage = twilioCli.messages.get(message.sid) >>> updatedMessage.status 'delivered' >>> updatedMessage.date_sent datetime.datetime(2015, 7, 8, 1, 36, 18)

  Entering message.sid show you this message’s long SID. By passing this SID to the Twilio client’s get() method ➊, you can retrieve a new Message object with the most up-to-date information. In this new Message object, the status and date_sent attributes are correct.

  The status attribute will be set to one of the following string values: 'queued', 'sending', 'sent', 'delivered', 'undelivered', or 'failed'. These statuses are self-explanatory, but for more precise details, take a look at the resources at http://nostarch.com/automatestuff/.

  Receiving Text Messages with Python

  Unfortunately, receiving text messages with Twilio is a bit more complicated than sending them. Twilio requires that you have a website running its own web application. That’s beyond the scope of this book, but you can find more details in the resources for this book (http://nostarch.com/automatestuff/).

  Project: “Just Text Me” Module

  The person you’ll most often text from your programs is probably you. Texting is a great way to send yourself notifications when you’re away from your computer. If you’ve automated a boring task with a program that takes a couple of hours to run, you could have it notify you with a text when it’s finished. Or you may have a regularly scheduled program running that sometimes needs to contact you, such as a weather-checkin
g program that texts you a reminder to pack an umbrella.

  As a simple example, here’s a small Python program with a textmyself() function that sends a message passed to it as a string argument. Open a new file editor window and enter the following code, replacing the account SID, auth token, and phone numbers with your own information. Save it as textMyself.py.

  #! python3 # textMyself.py - Defines the textmyself() function that texts a message # passed to it as a string. # Preset values: accountSID = 'ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' authToken = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' myNumber = '+15559998888' twilioNumber = '+15552225678' from twilio.rest import TwilioRestClient ➊ def textmyself(message): ➋ twilioCli = TwilioRestClient(accountSID, authToken) ➌ twilioCli.messages.create(body=message, from_=twilioNumber, to=myNumber)

  This program stores an account SID, auth token, sending number, and receiving number. It then defined textmyself() to take on argument ➊, make a TwilioRestClient object ➋, and call create() with the message you passed ➌.

  If you want to make the textmyself() function available to your other programs, simply place the textMyself.py file in the same folder as the Python executable (C:Python34 on Windows, /usr/local/lib/python3.4 on OS X, and /usr/bin/python3 on Linux). Now you can use the function in your other programs. Whenever you want one of your programs to text you, just add the following: