Creating a Visual Comparison Tool for Front-End Developers 🔬

Regressions are the worst, and they’re inevitable in any software development cycle. We can mitigate their occurances as much as possible through automated testing, but getting 100% test coverage is a time-consuming task and sometimes not viable. Something I’ve run into a lot as a Front-End developer are visual regressions, where a change will be made to a stylesheet that at first glance seems fine, but later causes chaos somewhere else in the application. As the length and complexity of a project increases, making sure your styles don’t regress can be a tricky task.

I’ve used a couple of tools in the past that show visual diffs between two pages, but I’ve always wondered what it would take to make my own version. So join me as I dive down the rabbit hole once again to create my own CRUD (Crete, Read, Update, Delete) application to do exactly that. For this project I’ll be using NodeJS, React and Redux.

Creating the API

The first order of business was setting up a database and creating an API to store the tests. To handle this I decided to utilize Express for my routes and MongoDB for the database. To get started I laid out the database connections using the Mongoose package, requiring the path from the /config/database.config.js file. Once the connection has been established I required the routes for the API and then started Express.

const express = require('express');
const bodyParser = require('body-parser');
const app = express();
const mongoose = require('mongoose');
const MongoConfig = require('./config/database.config');
const morgan = require('morgan');
const cors = require('cors');
const port = 9090;

app.use(morgan('combined'));
app.use(cors());
app.use(bodyParser.urlencoded({ extended: true }));
app.use(bodyParser.json());

mongoose.Promise = global.Promise;

// Establish connection to the database...
mongoose.connect(MongoConfig.url, { useNewUrlParser: true })
.then(() => {
  require('./routes/tests.routes.js')(app);

  app.listen(port, () => {
    console.log('Now listening on port ', port)
  })

}).catch(error => {
  console.error('Encountered an error while starting the API: ', error)
  process.exit();
});

Afterwards I started setting up the API endpoints. As I intend for this to be a CRUD application I started with the usual suspects, I needed an endpoint to create a test, retrieve all tests, find a specific one, update and delete. Inside of my routes/tests.route.js file I exported a module that defined these endpoints, and then tied them to a function that would later be defined within controllers/tests.controller.js.

const tests = require('../controllers/tests.controller.js')

module.exports = app => {
  app.post('/tests', tests.create);
  app.get('/tests', tests.findAll);
  app.get('/tests/:testId', tests.findOne);
  app.put('/tests/:testId', tests.update);
  app.delete('/tests/:testId', tests.delete);
}

I knew what type of information I’d need in the database in order to run a test, so the next step was creating a schema for the test object before we started interfacting with the database. I set this up within models/tests.model.js. The schema tells the database what fields it can expect. However, it doesn’t define which fields are required. I’ll set that up in our controller within the tests.create and tests.update methods.

const mongoose = require('mongoose')

const TestSchema = mongoose.Schema({
  name: String,
  description: String,
  live: String,
  dev: String,
  size: Number
})

module.exports = mongoose.model('Test', TestSchema);

Because the tests will rely on a Node service to run I’d need some sort of interface that would allow our Front-End to trigger a process on the backend. Therefore in addition to the endpoints I’ve already created I setup two more, one that will run all tests, and one that will run a specific test.

module.exports = app => {
  // ...
  app.get('/run', tests.testAll);
  app.get('/run/:testId', tests.testOne);
}

The intention here is to fire a Node function using the test data in the database when the endpoint is requested, generating the visual diff images. From an API perspective it would return a success or failure boolean when it’s done processing so we can inform our Front-End of the status. If this application was being deployed to a server there would need to be more considerations made for this in regards to performance and authentication, but for local use only it’s fine.

exports.testAll = (req, res) => {
  Test.find().then((tests) => {
    // We should be firing a function here to generate the diffs...
  })
}

exports.testOne = (req, res) => {
  const tests = []
  Test.findById(req.params.testId).then(test => {

    tests.push(test);

    /* This function should behave like the one in testAll, except the array will only contain one test.
    I'll be keeping them both arrays for consistency. */
  })
}

Generating the Diffs

To generate the diff images I used two Node libraries. One is Puppeteer, which provides a high-level API to control the Chrome browser; I’ll be using this to take the page screenshots. The other is Pixelmatch, a library by Mapbox which highlighlights pixel differences between two images. I started off by creating and exporting a module which would capture the initial browser shots. This module should accept an array of test data as its arguement and return the success flag when it was done creating the visual diff images on the localdisk. In order to achieve this I had the module return a promise.

async function runTest(test) {
  // This is the function that will actually run the test!
}

module.exports = tests => {
  return new Promise((resolve, reject) => {
    // This promise will resolve once we have our visual diffs
  })
}

As it’s expecting an array of data, I’ll need to fire off the runTest function for every item in the array, therefore I’ll need to lean even harder on the promise library. Using Array.map I converted my array of data into promises. That way I can then pass that mapped array into a Promise.all statement and resolve our wrapping promise once all of our items in the array are done processing.

I also created an if statement that would fire at the start of the block that will create our folder to hold the images in if it doesn’t exist already, if the user accidently deletes the folder it will error out the entire process and the application would no longer work.

const mkdirp = require('mkdirp');

module.exports = tests => {
  return new Promise((resolve, reject) => {
    
    // Makes sure that our save directory for our images actually exist
    if (!fs.existsSync(directory)) {
      mkdirp(directory, (() => {
        console.log('Created directory')
      }));
    };

    let testPromises = tests.map(test =>  {
      return runTest(test);
    });
  
    Promise.all(testPromises).then(results => {
      resolve(results);
    }).catch(() => {
      reject('Encountered processing error.');
    });
  })
}

The goal of the runTest function is to start a headless Chrome browser using Puppeteer and then do the following:

  • 1. Set the viewport width if a size value is provided.
  • 2. Navigate to the live page and wait for a few seconds to let any animations finish playing.
  • 3. Take a screenshot of the live page.
  • 4. Do the same for the dev page.
  • 5. Generate the pixel comparison overlay using the live and dev page images.
  • 6. Close the browser and return.

The documentation for Puppeteer is quite clear, and getting an initial proof of concept going was quite simple.

async function timeout(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

async function runTest(test) {
  const { _id, name, live, dev, size } = test;

  // Launches the browser.
  const browser = await puppeteer.launch();

  // Creates a new 'tab'.
  const page = await browser.newPage();

  /* If a size is present we set the viewport, otherwise we leave it
  open to the default setting. */
  if (size) {
    await page.setViewport({
      width: size,
      height: 0
    })
  };

  /* Navitates puppeteer to the page to take the screenshot. */
  await page.goto(live)
      
  /* We set a timeout here to make sure that all initial load animations
  have finished playing before we take the screenshots. */
  await timeout(5000);

  // Takes the screenshot for the live page.
  await page.screenshot({
    path: `${directory}/live_${_id}.png`,
    fullPage: true
  });

  // Navigates to the dev page.
  await page.goto(dev)

  await timeout(5000);

  // Takes the screenshot for the dev page.
  await page.screenshot({
    path: `${directory}/dev_${_id}.png`,
    fullPage: true
  });

  /* Once we have both of our screenshots we send them over to the compare function
  so we can get a pixel overlay for each image. */
  await compare(_id);

  // Closes the browser.
  await browser.close()

  // Sets 'success': true in our test object and returns.
  test['success'] = true;
  return test;
}

This initial pass of the function does mostly everything I need it to, except there’s a problem, primarily in the error handling department. If I run through this function and provide it two valid paths it will complete and everything will be fine, however if I provide it at least one invalid path, it will error and not pass back any form of usable data for our API response. To improve this I utilized try, catch and finally to throw errors when they are encountered. In the example below I setup catch cases on the goto calls, and then throw an error if it’s triggered, moving it to the wrapping catch case, this allows me to set the property success to false. If it reaches the end of the try case then finally is triggered so I can then set success to true.

async function runTest(test) {
  const { _id, name, live, dev, size } = test;
  const browser = await puppeteer.launch();

  try {
    const page = await browser.newPage();

    /* If a size is present we set the viewport, otherwise we leave it
    open to the default setting. */
    if (size) {
      await page.setViewport({
        width: size,
        height: 0
      })
    };

    /* If an error is encountered we throw an exception cancelling the rest of the process.
    These errors are collected in an API response so we can alert the front-end. */
    await page.goto(live).catch((error) => {
      throw `${error} ${live}`
    });

    /* We set a timeout here to make sure that all initial load animations
    have finished playing before we take the screenshots. */
    await timeout(5000);

    await page.screenshot({
      path: `${directory}/live_${_id}.png`,
      fullPage: true
    });

    await page.goto(dev).catch((error) => {
      throw `${error} ${dev}`
    });

    await timeout(5000);

    await page.screenshot({
      path: `${directory}/dev_${_id}.png`,
      fullPage: true
    });

    /* Once we have both of our screenshots we send them over to the compare function
    so we can get a pixel overlay for each image. */
    await compare(_id);

  } catch(error) {
    await browser.close();
    test['success'] = false;
    return test;
  } finally {
    await browser.close();
    test['success'] = true;
    return test;
  }
}

The compare function that gets fired near the end of runTest also resolves a promise when it’s done processing, signalling the async function to move onto the next step. The code is mostly unchanged from the Pixelmatch README example.

module.exports = id => {
  return new Promise((resolve, reject) => {
    const directory = DirectoryConfig.path;
    let filesRead = 0;
  
    const live = fs.createReadStream(`${directory}/live_${id}.png`).pipe((new PNG()).on('parsed', doneReading));
    const dev = fs.createReadStream(`${directory}/dev_${id}.png`).pipe((new PNG()).on('parsed', doneReading));
  
    function doneReading() {
      if (++filesRead < 2) return;
    
      const diff = new PNG({width: live.width, height: live.height})
      pixelmatch(live.data, dev.data, diff.data, live.width, live.height, { threshold: 0.1 });
    
      diff.pack().pipe(fs.createWriteStream(`${directory}/diff_${id}.png`));
      resolve('Resolving')
    }
  })
}

With these building blocks in place I can now call the capture module when our endpoint is requested, and trigger a response with usable data. The response will be delayed until the runTest function has been fired for every item in the array.

const capture = require('../utilities/capture');

exports.testAll = (req, res) => {
  Test.find().then((tests) => {
    capture(tests).then(testResults => {
      res.send(testResults);
    }).catch(error => {
      res.send(error);
    })
  })
}

exports.testOne = (req, res) => {
  const tests = []
  Test.findById(req.params.testId).then(test => {

    tests.push(test);

    capture(tests).then(testResults => {
      res.send(testResults)
    }).catch(error => {
      res.send(error)
    })
  })
}

If you’d like to see the other API controllers you can view them on GitHub here. The delete function also interacts with the localdisk and deletes screenshots from tests which have been previously run when the delete endpoint is requested.

Front-End

For the Front-End I decided to make a simple application using React and Redux that will display a list of tests, with a test page to display our diffs, and a form page to create/edit the tests. Using Redux and Axios I began by creating a series of action creators that will fire off the API calls to the Node service I just created.

import axios from 'axios';
import {
  FETCH_TEST,
  FETCH_TESTS,
  EDIT_TEST,
  RUN_TEST,
  ADD_TEST,
  REMOVE_TEST,
} from './types';

// The API runs on a different port from the Front-End
export const ROOT_URL = 'http://localhost:9090';

/**
* @desc Fetches all available tests from the API.
**/
export function fetchTests() {
  const request = axios.get(`${ROOT_URL}/tests`);

  return {
    type: FETCH_TESTS,
    payload: request
  };
}

/**
* @desc Fetches a test from the API based on an id value.
* @param {string} id - The id of the test.
**/
export function fetchTest(id) {
  const request = axios.get(`${ROOT_URL}/tests/${id}`);

  return {
    type: FETCH_TEST,
    payload: request
  };
}

/**
* @desc Runs a specific test stored in the API based on an id value.
* @param {string} id - The id of the test.
**/
export function runTest(id) {
  const request = axios.get(`${ROOT_URL}/run/${id}`);

  return {
    type: RUN_TEST,
    payload: request
  };
}

/**
* @desc Adds a test to the API.
* @param {object} props - An object containing the test data.
**/
export function addTest(props) {
  const request = axios.post(`${ROOT_URL}/tests`, props);

  return {
    type: ADD_TEST,
    payload: request
  };
}

/**
* @desc Edits a specific test stored in the api.
* @param {object} props - An object containing the test data.
* @param {string} id - The id of the test.
**/
export function editTest(props, id) {
  const request = axios.put(`${ROOT_URL}/tests/${id}`, props);

  return {
    type: EDIT_TEST,
    payload: request 
  };
}

/**
* @desc Removes a test from the API.
* @param {string} id - The id of the test.
**/
export function removeTest(id) {
  const request = axios.delete(`${ROOT_URL}/tests/${id}`);

  return {
    type: REMOVE_TEST,
    payload: request
  };
}

Some of the action creators I created require a reducer to bind the response to the Redux state, so in the reducers/index.js file I added some. In the below example all represents all of the tests in the API /tests endpoint, whereas test represents the current test that is being viewed. testValidation will display the result of a test that has just finished running, and will allow us to access the success flag from the API response.

import { FETCH_TESTS, FETCH_TEST, RUN_TEST } from '../actions/types';

const INITIAL_STATE = { all: [], test: null, testValidation: null };

export default function(state = INITIAL_STATE, action) {
  switch(action.type) {

  case FETCH_TEST:
    return { ...state, test: action.payload.data };

  case FETCH_TESTS:
    return { ...state, all: action.payload.data }

  case RUN_TEST:
    return { ...state, testValidation: action.payload.data }

  default:
    return state;
  }
}

With the action creators and reducers setup I can utilize the Redux connect method to bind the Redux state to the React component props. In the example below I fire off the fetchTests action creator before the component mounts within componentWillMount(), and then using connect() I pull the state from Redux and utilize the mapStateToProps function to bind that state to our component props, allowing me to access this.props.tests within the render() method.

I’ve simplified the following React examples to show only the relevent information. You can view the full code on GitHub.

import React, { Component } from 'react';
import PropTypes from 'prop-types';
import { connect } from 'react-redux';
import { fetchTests } from '../actions/index';
import { Link } from 'react-router';

class TestIndex extends Component {
  componentWillMount() {
    this.props.fetchTests();
  }

  renderTests() {
    const { classes } = this.props

    return this.props.tests.map(test => {
      return (
        <Grid item xs={12} sm={6} key={test._id}>
            <Card className={classes.card}>
              <CardContent>
                <Typography variant="headline" component="h2">
                <Link to={`/tests/${test._id}`} style=>{test.name}</Link>
                </Typography>
                <Typography className={classes.pos} color="textSecondary">
                  {test.description}
                </Typography>
              </CardContent>
            </Card>
        </Grid>
      )
    })
  }

  render() {
    const { classes } = this.props;

    return (
      <div className={classes.root}>
        <Grid container spacing={24}>
          {this.props.tests.length > 0  ? 
            this.renderTests() : 
            <div>There are no tests to show!</div>
          }
        </Grid>
      </div>
    )
  }
}

function mapStateToProps(state) {
  return { tests: state.tests.all }
}

TestIndex.propTypes = {
  classes: PropTypes.object.isRequired
};

export default withStyles(styles)(connect(mapStateToProps, { fetchTests })(TestIndex));

As there needs to be a page to view a specific test, I’ll need to do something similar to the example above. The primary change being that I’ll need to run fetchTest instead using the id in the page route as the arguement. As I pushed the user to the /tests/:id path within TestIndex using react-router, I can harness that data by accessing this.props.params.id. You can learn more about how react-router works here.

class TestShow extends Component {
  static contextTypes = {
    router: PropTypes.object
  };

  constructor(props) {
    super(props)

    this.state = {
      running: false,
      snackbar: false,
      error: ''
    }
  }

  componentWillMount() {
    this.props.fetchTest(this.props.params.id);
  }
  

  render() {
    const { classes, test } = this.props;

    if (!test) {
      return <LinearProgress />
    }

    // ...
  }
}

I also need to create a way to actually run the test, and delete them if desired. Just as I’ve run fetchTest, I can extend the same functionality to use our other action creators on button presses. You can see in the onRunClick() method that I’m calling the runtest action creator, and then checking in the success handler if the first index of testValidation has success: false, this way i can display an error to the user if there was an issue with the test. I can also use the this.state.running boolean to display a loading bar while the test runs for a better user experience.

class TestShow extends Component {
  static contextTypes = {
    router: PropTypes.object
  };

  constructor(props) {
    super(props)

    this.state = {
      running: false,
      snackbar: false,
      error: ''
    }
  }

  componentWillMount() {
    this.props.fetchTest(this.props.params.id);
  }


  onRemoveClick() {
    this.props.removeTest(this.props.params.id)
      .then(() => {
        this.context.router.push('/');
      })
  }

  onEditClick() {
    this.context.router.push(`/tests/${this.props.params.id}/edit`);
  }

  onRunClick() {
    this.props.runTest(this.props.params.id).then(result => {
      /* Checks to see if the validation wasn't succesful. If it's not
      we display an error prompt. */
      if (!this.props.testValidation[0].success && this.state.running) {
        this.setState({
          running: false,
          snackbar: true,
          error: 'There was an issue running the test, please edit the test and try again.'
        })
      }
    })

    // Resets the error prompt everytime the button is pressed.
    this.setState({
      running: true,
      snackbar: false,
      error: ''
    })
  }

  render() {
    const { classes, test } = this.props;

    if (!test) {
      return <LinearProgress />
    }

    // ...
  }
}

I also needed a component to display the visual diffs as that’s what this entire application is all about! For this I created a simple component which accepted the src and overlay paths as props, which get handed down by the TestShow component. I then setup a simple on click handler which toggles the image source to the overlay. If there’s an error with the test, for instance if the test fails to run or hasn’t been run yet at all, then the onError handler is triggered setting the image to the placeholer.

class Diff extends Component {
  constructor(props) {
    super(props)

    this.state = {
      showOverlay: false
    }

    this.handleClick = this.handleClick.bind(this)
  }

  handleClick() {
    this.setState({
      showOverlay: !this.state.showOverlay
    })
  }

  render() {
    const { classes } = this.props;

    return (
      <Card className={classes.card}>
        <CardContent>
          <Typography variant="headline" component="h2">
            Dev
          </Typography>
          <Typography className={classes.pos} color="textSecondary">
            {this.props.path}
          </Typography>
          <img 
            className={classes.img} 
            src={this.state.showOverlay ? this.props.overlay : this.props.src}
            alt="Screenshot of the dev site."
            onClick={this.handleClick}
            onError={(e) => {e.target.src="/error.png"}}
            style= />
        </CardContent>
      </Card>
    )
  }
}

The last order of business is creating a form which can create and edit the test data, and submit it to the database. As I already have an action creator for this I just need to push an object into the function. I decided to use the redux-form library for this, as it has some great built-in tools for form validation. Whenever we touch our text boxes the validate function should get run which checks to make sure that we’re not leaving any required fields empty. There’s also some checks setup on the backend that require the name, live and dev fields to be filled out, otherwise the endpoint will reject the request.

class TestForm extends Component {
  renderTextField = ({
    input,
    label,
    placeholder,
    required,
    defaultValue,
    meta: { touched, error },
    ...custom
  }) => (
    <TextField
      label={touched && error ? error : label}
      error={touched && error ? true : false}
      required={required}
      placeholder={placeholder}
      {...input}
      {...custom}
    />
  )

  render() {
    const { classes } = this.props

    return (
      <div className={classes.root}>
        <Grid container spacing={24}>
          <Grid item xs={12}>
            <Card>
              <form onSubmit={this.props.onSubmit}>
                <CardContent>
                  <Typography variant="headline" component="h2">
                    {this.props.title}
                  </Typography>
                  <div>
                    <Field 
                      component="input" 
                      type="text" 
                      name="name"
                      label="Name"
                      margin="normal"
                      placeholder="The name of your test"
                      className={classes.textField}
                      component={this.renderTextField} />
                  </div>

                  <div>
                    <Field 
                      component="input" 
                      type="text" 
                      name="description"
                      label="Description"
                      margin="normal"
                      placeholder="Describe your test."
                      className={classes.textField}
                      component={this.renderTextField}  />
                  </div>

                  <div>
                    <Field 
                      component="input" 
                      type="text" 
                      name="live"
                      label="Live URL"
                      margin="normal"
                      placeholder="The url path for your live site."
                      className={classes.textField}
                      component={this.renderTextField} />
                  </div>

                  <div>
                    <Field 
                      component="input" 
                      type="text" 
                      name="dev"
                      label="Dev URL"
                      margin="normal"
                      placeholder="The url path for your dev site."
                      className={classes.textField}
                      component={this.renderTextField} />
                  </div>

                  <div>
                    <Field 
                      component="input" 
                      type="number" 
                      name="size"
                      label="Browser Size"
                      margin="normal"
                      placeholder="The browser width you'd like your test to run at."
                      className={classes.textField}
                      component={this.renderTextField} />
                  </div>
                </CardContent>
                <CardActions>
                  <Button type="submit">Submit</Button>
                  <Link to="/" style=><Button type="submit">Cancel</Button></Link>
                </CardActions>
              </form>
            </Card>
          </Grid>
        </Grid>
      </div>
    )
  }
}

export function validate(values) {
  const errors = {};

  if (!values.name) {
    errors.name = 'Enter a name for your test...'
  };

  if (!values.live) {
    errors.live = 'Live URL is required...'
  };

  if (!values.dev) {
    errors.dev = 'Dev URL is required...'
  };

  return errors;
}

TestForm.propTypes = {
  classes: PropTypes.object.isRequired,
  onSubmit: PropTypes.func.isRequired
};

export default withStyles(styles)(TestForm);

I wanted this component to be reusable for both the new test and edit use cases, so I set it up to accept an onSubmit function as a prop, which will be our action creator that will submit the form data to the database. You can see in the export statement that I’m setting up the Redux form here, and even requiring validate from the TestForm component so it knows to run it.

import React, { Component } from 'react';
import PropTypes from 'prop-types';
import { reduxForm } from 'redux-form';
import { connect } from 'react-redux';
import { addTest } from '../actions/index';
import TestForm, { validate } from './TestForm';

class TestNew extends Component {
  static contextTypes = {
    router: PropTypes.object
  };

  onSubmit(props) {
    this.props.addTest(props)
      .then(() => {
        this.context.router.push('/');
      })
  }

  render() {
    const { handleSubmit } = this.props

    return (
      <TestForm 
        onSubmit={handleSubmit(this.onSubmit.bind(this))} 
        title="Create a Test" />
    )
  }
}

export default reduxForm({
  form: 'TestsNewForm',
  fields: ['name', 'description', 'live', 'dev', 'size'],
  validate
})(
	connect(null, { addTest })(TestNew)
);

With that setup I couldnow include the <TestNew />or <TestEdit /> component in a route and have them reuse the same form with varying functionality. The primary difference between the two of them is that TestEdit starts off with some initial state that is sourced from the test that is currently being edited.

In Conclusion

After numerous amounts of testing and making sure everything worked together, I finally ended up with an application that worked! This post was intended to give you insight into my thought process while building this project, however if you’d like to go into more detail you can check out the source on Github.

As always if you have any questions or feedback feel free to leave a comment, or reach out on Twitter or via my contact form.

  • Share

  • Share on Facebook
  • Tweet
  • Submit to Reddit
  • Share on LinkedIn
  • Send email

Comments 💬