Server Testing via Visual Regression Testing
One of my least favorite tasks at work is testing new production servers. The team that provisions them doesn’t use any automated tools and manually sets up each one by hand. Unfortunately, this usually leads to missed settings and configurations, so we have to manually test several of our important systems on each server before signing off on them.
While I was working on it this time, I started thinking about ways that I could automate the process on our end. It basically is just browsing to a specific list of web pages we maintain on multiple servers and verifying everything looks the same on our current in-use production servers. Thinking about it, it sounded awfully similar to visual regression testing.
So I wrote up a quick node.js program that takes an array of servers to compare, an array of pages you want to compare, and the server you want to compare against. First, it generated screenshots for each page on the “baseline” server. Then, it generates a screenshot of each page on every other server and compares them to the baseline using the Pixelmatch library. If it finds more than 10 pixels of difference in the screenshot, it will report out to the console and generate a “Diff” screenshot that highlights the differences.
Here is what it looks like when searching for “tea” on both google.com and google.de.
www.google.com
www.google.de
The results
For GVSU, I can now use this to verify all the servers in a few seconds instead of a few minutes. I can check our homepage, pages from the cms we manage, and some custom pages I created just for this. For example, there often are issues with a charting library we use, so I created a page that is just a static chart that I can now hit with the automated tool to verify they are working.
I threw the script up on Github in case it might be useful for anyone else: https://github.com/brentswisher/visual-server-tester. If you find it useful, I’d love to hear about it!