Python Automation Tools: Tools of the Trade

Jul 2, 2020

Python is an extremely powerful tool that allows you to quickly write scripts to perform the desired tasks and bring automation into different domains of life. Python has a large ecosystem of libraries and tools and sometimes it can be overwhelming to find and choose the right tools for the job. So let’s discuss some of the most commonly used and loved Python tools used in the field of automation. In case you are new to Python or want to master the fundamentals of the language you may want to check out the article on best books to learn automation with Python.

Python Automation Tools for the Web

Requests

requests is an amazing library which massively simplifies sending HTTP requests to any HTTP server via Python. Using requests library alone can get you very far and help you automate
many things. For example if you had a web app hosted on the free tier of Heroku where the app goes to sleep after half an hour of idle period, you could send a request every half an hour to stop the web app from going to sleep. requests library is very commonly used for accessing the REST API provided by other services to make use of their offerings. For example you can use requests to automatically upload files to Dropbox, send messages in a Discord channel, upload files to Google Drive and so on.

BeautifulSoup

Generally requests and BeautifulSoup are used together. Whereas requests is used to send HTTP requests to servers and receive the response, BeautifulSoup allows you to extract the needed information from that response. BeautifulSoup is an HTML parser which can analyze an HTML document and lets you execute ad-hoc queries on it to get the desired information, such as fetching all the texts within the paragraph tags of class ‘item’, fetching all the links or image urls in a page and so on. BeautifulSoup lets you use different ways of querying for the data in HTML document, eg. using CSS selectors, HTML attributes such as id, class or tag, and so on.

Selenium

While requests fetches the static response of an HTTP request, it’s not capable of executing the JavaScript code present in the response. In the modern web JavaScript is used excessively in most kinds of websites and often the desired action on a website can’t be automated without running those scripts. However there’s no practical way of running those scripts and providing those scripts the environment they need to run properly without spawning a browser instance. This is where Selenium comes into picture. Using Selenium you can spawn a browser instance such as that of Chrome or Firefox and programmatically inject keypresses, clicks, scrolls or other actions within the browser to automatically surf a website and perform the actions needed on it to accomplish your tasks.

Python Automation Tools for Desktop

PyAutoGUI

While the libraries mentioned above are very powerful tools to automate web related tasks, there are sometimes cases when you need to automate parts of the GUI of your computer. Or may be the web app you’re targeting is highly graphics-dominated and you decided to spawn a browser from Selenium and set up the environment, and then work on the pixels of the website. Or you might wish to automate the paint application in your computer and draw images automatically using the app’s widgets just for fun. Also some cases can arise where a native application doesn’t provide any CLI or programmable API and interacting with it graphically is the only way to automate tasks on that app. In all these cases PyAutoGUI can be a very useful tool to analyze the pixels on a screen and automatically supply clicks, keystrokes and so on in the desired manner. Also this library is written by Al Sweigart himself, the author of the famous book in Python community, Automate the Boring Stuff with Python.

Celery

Celery is a powerful library often used in web development and other areas to run tasks asynchronously or to schedule tasks in future or periodically. You can use Celery to run a task periodically at a specified time eg. every Monday at 9 PM, or at every one hour interval and so on. This feature can be very helpful if you want to automate tasks such as taking periodic backups, checking for changes in your Google Drive/Dropbox storage, or sending alerts for new tweets containing a hashtag and so on.

Python automation tools for sending notifications

When executing long-running Python programs or running programs in the background, you’ll soon get to realize the importance of sending notifications to yourself regarding the status of your programs when important events take place. For example your programs can throw an error and fail due to a magnitude of reasons you couldn’t have predicted and in these cases it’s absolutely great to have the ability to be notified of the issue as well as the traceback and context of the program which led to the problem. There are different ways to let your Python script send notifications to you and let’s discuss about some of them.

Email

Probably the simplest as well as an effective way to send notifications to yourself or your team from your programs is via email. It’s pretty straightforward to send email to yourself and also doesn’t require much setup to get going quickly with this system. All you need is an email account and some details of the email provider and you’re good to go using Python’s builtin smtplib and other libraries!

Discord/Telegram

You can also send yourself notifications on a Discord channel very easily using Discord’s webhooks using the requests library we discussed earlier. You can also create a highly interactive interface for your program via Discord by creating a Discord bot using a library such as discord.py. Likewise there are similar tools for Telegram which you can use to send messages to you or create interactive Telegram bots with. Using these tools you can easily send yourself rich and well-formatted alerts about different kinds of events.

Deploying your Python Automation Tools to the Cloud

Oftentimes the scripts you write to automate your tasks require running continuously or at specific times, or have other requirements such as good internet connectivity and so on. Depending on the needs it may be difficult or impossible to achieve that on your personal computer. In these cases it’s a good idea to deploy your script on the cloud and run it there. DigitalOcean is one of the best cloud based virtual server (VPS) providers in the market with a simple and developer-friendly platform to provision a server instance and get going. Other relevant providers in the market include Linode, AWS, GCP, Azure, Heroku and PythonAnywhere. Some of these services provide a free tier as well for hobby projects which might be sufficient for your needs.

Python Automation Tools for Testing

Testing is a great tool that helps developers write reliable, consistent and efficient software. In Python testing is especially important given its duck typing and interpreted nature where some of the more obvious bugs which would have been detected in the compile time in compiled languages can evade detection and lead to errors in deployment. However given its duck typing and other productivity-boosting features it also means that writing tests in Python is a faster and less painful process than it might be in more rigid languages. It’s also important to note that testing is more than just a tool and there are entire ideologies on software development in general which put testing at their core. Test-driven development is one such methodology. If you haven’t, I highly recommend reading the book Test-Driven Development with Python to learn more about testing and other good practices. Python has a great ecosystem when it comes to writing and running tests in a systematic and comprehensive manner and let’s discuss some of them.

unittest

unittest is a builtin library in Python which can take you quite far in your testing. It has a relatively lower learning curve than many other testing libraries and thus easier to start using, but is powerful enough to serve the purpose feasably in most cases.

pytest

pytest is currently probably the most popular Python testing framework. It has a more succint style and has more powerful features than unittest. It lets you write more compact and better organized tests and provides many excellent features that are needed in modern software development to write and run effective test suites.

tox

tox is a more advanced testing framework which lets you run your tests in multiple environments which can be very useful if you’re going to package and distribute your Python application to other users who might be using different operating systems, Python versions and so on.

These are some of the Python automation tools which often come very handy when automating different kinds of tasks. However the Python ecosystem offers many more amazing and well-designed libraries and tools which can help you to automate different kinds of tasks. If you haven’t already, I highly recommend reading the Part 2 of the book Automate the Boring Stuff with Python (Part 1 as well if you are new to Python and want to master the basics!). The book provides a great introduction to automating a wide range of tasks using Python and will certainly get your mind coming up with new ideas and ways in which you could automate some of the tasks you perform in your daily life, at work and so on.