Python is an extremely powerful tool that allows you to quickly write scripts to perform the desired tasks and bring automation into different domains of life. Python has a large ecosystem of libraries and tools and sometimes it can be overwhelming to find and choose the right tools for the job. So let’s discuss some of the most commonly used and loved Python tools used in the field of automation. In case you are new to Python or want to master the fundamentals of the language you may want to check out the article on best books to learn automation with Python.
requests is an
amazing library which massively simplifies sending HTTP requests to any HTTP
server via Python. Using requests library alone can get you very far and help
many things. For example if you had a web app hosted on the free tier of Heroku where the app goes to sleep after half an hour of idle period, you could send a request every half an hour to stop the web app from going to sleep.
requests library is very
commonly used for accessing the REST API provided by other services to make
use of their offerings. For example you can use
automatically upload files to Dropbox, send messages in a Discord channel,
upload files to Google Drive and so on.
used together. Whereas
requests is used to send HTTP requests to
servers and receive the response,
BeautifulSoup allows you to
extract the needed information from that response.
BeautifulSoup is an HTML parser which can analyze an HTML
document and lets you execute ad-hoc queries on it to get the desired
information, such as fetching all the texts within the paragraph tags of class
‘item’, fetching all the links or image urls in a page and so on.
BeautifulSoup lets you use different ways of querying for the
data in HTML document, eg. using CSS selectors, HTML attributes such as id,
class or tag, and so on.
requests fetches the static response of an HTTP request,
websites and often the desired action on a website can’t be automated
without running those scripts. However there’s no practical way of
running those scripts and providing those scripts the environment they need to
run properly without spawning a browser instance. This is where
Selenium comes into
picture. Using Selenium you can spawn a browser instance such as that of
Chrome or Firefox and programmatically inject keypresses, clicks, scrolls or
other actions within the browser to automatically surf a website and perform
the actions needed on it to accomplish your tasks.
While the libraries mentioned above are very powerful tools to automate web related tasks, there are sometimes cases when you need to automate parts of the GUI of your computer. Or may be the web app you’re targeting is highly graphics-dominated and you decided to spawn a browser from Selenium and set up the environment, and then work on the pixels of the website. Or you might wish to automate the paint application in your computer and draw images automatically using the app’s widgets just for fun. Also some cases can arise where a native application doesn’t provide any CLI or programmable API and interacting with it graphically is the only way to automate tasks on that app. In all these cases PyAutoGUI can be a very useful tool to analyze the pixels on a screen and automatically supply clicks, keystrokes and so on in the desired manner. Also this library is written by Al Sweigart himself, the author of the famous book in Python community, Automate the Boring Stuff with Python.
is a powerful library often used in web development and other areas to run
tasks asynchronously or to schedule tasks in future or periodically. You can
Celery to run a task periodically at a specified time eg.
every Monday at 9 PM, or at every one hour interval and so on. This feature
can be very helpful if you want to automate tasks such as taking periodic
backups, checking for changes in your Google Drive/Dropbox storage, or sending
alerts for new tweets containing a hashtag and so on.
When executing long-running Python programs or running programs in the background, you’ll soon get to realize the importance of sending notifications to yourself regarding the status of your programs when important events take place. For example your programs can throw an error and fail due to a magnitude of reasons you couldn’t have predicted and in these cases it’s absolutely great to have the ability to be notified of the issue as well as the traceback and context of the program which led to the problem. There are different ways to let your Python script send notifications to you and let’s discuss about some of them.
Probably the simplest as well as an effective way to send notifications to
yourself or your team from your programs is via email. It’s pretty
straightforward to send email to yourself and also doesn’t require much
setup to get going quickly with this system. All you need is an email account
and some details of the email provider and you’re good to go using
smtplib and other libraries!
You can also send yourself notifications on a Discord channel very easily
using Discord’s webhooks using the
requests library we
discussed earlier. You can also create a highly interactive interface for your
program via Discord by creating a Discord bot using a library such as
there are similar tools for Telegram which you can use to send messages to you
or create interactive Telegram bots with. Using these tools you can easily
send yourself rich and well-formatted alerts about different kinds of events.
Oftentimes the scripts you write to automate your tasks require running continuously or at specific times, or have other requirements such as good internet connectivity and so on. Depending on the needs it may be difficult or impossible to achieve that on your personal computer. In these cases it’s a good idea to deploy your script on the cloud and run it there. DigitalOcean is one of the best cloud based virtual server (VPS) providers in the market with a simple and developer-friendly platform to provision a server instance and get going. Other relevant providers in the market include Linode, AWS, GCP, Azure, Heroku and PythonAnywhere. Some of these services provide a free tier as well for hobby projects which might be sufficient for your needs.
Testing is a great tool that helps developers write reliable, consistent and efficient software. In Python testing is especially important given its duck typing and interpreted nature where some of the more obvious bugs which would have been detected in the compile time in compiled languages can evade detection and lead to errors in deployment. However given its duck typing and other productivity-boosting features it also means that writing tests in Python is a faster and less painful process than it might be in more rigid languages. It’s also important to note that testing is more than just a tool and there are entire ideologies on software development in general which put testing at their core. Test-driven development is one such methodology. If you haven’t, I highly recommend reading the book Test-Driven Development with Python to learn more about testing and other good practices. Python has a great ecosystem when it comes to writing and running tests in a systematic and comprehensive manner and let’s discuss some of them.
unittest is a builtin library in Python which can take you quite far in your testing. It has a relatively lower learning curve than many other testing libraries and thus easier to start using, but is powerful enough to serve the purpose feasably in most cases.
pytest is currently probably the most popular Python testing framework. It has a more succint style and has more powerful features than unittest. It lets you write more compact and better organized tests and provides many excellent features that are needed in modern software development to write and run effective test suites.
tox is a more advanced testing framework which lets you run your tests in multiple environments which can be very useful if you’re going to package and distribute your Python application to other users who might be using different operating systems, Python versions and so on.
These are some of the Python automation tools which often come very handy when automating different kinds of tasks. However the Python ecosystem offers many more amazing and well-designed libraries and tools which can help you to automate different kinds of tasks. If you haven’t already, I highly recommend reading the Part 2 of the book Automate the Boring Stuff with Python (Part 1 as well if you are new to Python and want to master the basics!). The book provides a great introduction to automating a wide range of tasks using Python and will certainly get your mind coming up with new ideas and ways in which you could automate some of the tasks you perform in your daily life, at work and so on.