Introducing Night Patrol

In the fast moving technology stacks of JavaScript, Nightwatch has grown to become one of the most popular front-end end-to-end testing frameworks around. It's an all-in-one solution allowing you to write tests without additional dependencies and effortlessly run them either locally or remotely.

We've been using Nightwatch at Unruly for a year now, and while it's not perfect, it's proven to be a lot more convenient than similar frameworks.

One of the problems we've had with Nightwatch is related to how we debug tests. During deployment, we'd run tens of tests and only one may fail. Finding this failing test and running it in isolation has always been tedious and time consuming since there's no IDE integrations to easily jump around the code base from Nightwatch's output.

This is where Night Patrol comes in. In the simplest sense, Night Patrol does what we would've done manually. It keeps track of various inputs sent to Nightwatch and keeps track of the resulting outputs. In this article, I'll be going through how Night Patrol works and how it's used.

Configuring Inputs to Nightwatch

Night Patrol essentially manages how inputs are sent to Nightwatch and provides a CLI to the user. Before we get into using Night Patrol, let's go through how Nightwatch is configured.

Configuration Files

Nightwatch uses a single configuration file to set itself up. We use this to tell Nightwatch where to find test files, and configure the various browsers to run the tests in (see Appendix [1] at the end for an example configuration). The properties we're interested in are:

• src_folders - where to find the test files
• test_settings - which environments to run the tests in.

Command-Line Flags

Along with a configuration files, Nightwatch's CLI takes various commands so you can pick out which tests to run:

• env - the environment to run in (defaults to all)
• test - the test file to run (defaults to all in the env)
• testcase - the test case to run (defaults to all in the test)

So a typical run can look like the following:

nightwatch --env chromeHeadless --test ./tests/scenarios/loadHomePage.js --testcase 'should load home page'

It's a Hierarchy

From these inputs, we can see that Nightwatch's tests are ordered in a sort of hierarchy. We have folders (i.e. directories), which include test files which in turn have test cases, all of which are run in specific environments.

The Problem

Since the test values are tied to src_folders, the user needs to know both where the test files are and which test cases are available when they want to run a single test case. This is because each testcase is unique to specific test files and aren't globally unique.

Failing tests may not be tied to your current changeset, so you may not be aware of their exact location without looking around.

On a small scale this isn't a huge problem, but as you add more test files, test cases and organise your test files into directories, it can become overwhelming.

Enter Night Patrol

Night Patrol solves this issue by providing contextual auto-completion. Rather than relying on you to remember how Nightwatch is configured, Night Patrol can parse Nightwatch's configuration and present a command-line interface for it. This allows the you to see what's available one step at a time.

Running Tests

Instead of using test files, Night Patrol can generate test suites using src_folders as the root, and the relative location of the test file as the suite name. Once we know the test file, we know which testcases are available.

Walking through the recording above:

• Switch to the env we want to run in. Similar to switching branches in Git.
• Select a test. Similar to switching directories.
• Run a testcase.

Handling Failures

When tests fail, Night Patrol keeps both the test scenario and test cases to re-run at any context.

Getting Help

Night Patrol itself is pretty straight forward, if you're ever lost, typing help will show the available commands in the current context.

You can also navigate history using the arrow keys.

Behind the Scenes

The project itself is very simple so I won't go into it in great depth. It started as a simple CLI using Vorpal.js, a framework for building session-based CLIs.

From there, as the feature set increased, I realised the correlation between the project and the many single-page applications I've written. So I introduced Redux to manage the state, which really opened my mind. I managed to do a ton of refactoring which made the code more readable and allowed me to easily introduce env switching since the state was global and the line between configuration and execution was clearly separated.

I briefly looked into adding routing to manage the current context in Vorpal and potentially using React to compose various features. A similar project exists for Blessed bindings, called react-blessed, but it doesn't provide the features which Vorpal does.

Binding React is definitely something I'll look into deeper when the feature set becomes complicated enough to warrant the investment.

The project is open source, so you can look through the code on GitHub.

Intentional Limitations

Night Patrol's workflow was specifically made to solve how we used Nightwatch at Unruly. We typically run Nightwatch against all tests, then we drill down to individual failing test cases. Because of this, Night Patrol doesn't support these features, since they weren't needed:

• You cannot run multiple tests in multiple environments.
• You cannot run all failures.
• Suites are one-level deep. So you cannot run a batch of tests at a directory-level.
• New test files and configurations aren't reloaded. You'll need to close and run Night Patrol again to refresh it.

If you need these features, feel free to submit an issue or pull request to the project.

Conclusions

I've learnt a lot from this project, here are a few of my takeaways.

Developer Interfaces Matter

A lot of the time, I feel as developers we often work hard on interfaces for our users, but neglect the interfaces we use every minute.

By using my knowledge of Single-Page Web Applications and the frameworks around it, I was able to easily create a command-line interface to solve a development issue. One that I can use and see being used on a day-to-day basis as part of my workflow. It has both improved my understand on how we work as developers and how frameworks, like Redux, can be used in a variety of applications and not just web interfaces.

Going forward, I hope to create similar tools to improve my workflow without relying on rote memorisation every time a new tool is introduced into the stack.

CLIs vs IDEs

One of the decisions I had to make early on was choosing between making a CLI and making a Plugin for WebStorm. While there's a huge convenience in having the tests run in WebStorm along side its other compatible test runners (like Mocha), I didn't want to invest too much time in learning WebStorm's Plugin SDK and getting side-tracked. Also, as the JavaScript community is a bit split in terms of IDEs, I didn't want to restrict the solution to a single product.

In the future, I'll continue to prefer CLIs over IDEs for the same reasons. The CLI is forever, IDEs come and go.

Separating Frameworks from Interfaces

There are already many frameworks that have separated the core of their framework from their user interface. A huge benefit in an ecosystem like Node.js, which seems to change every other week. Rather than learning individual interfaces for every framework, we can continue using the interfaces we're familiar with and switch frameworks, reducing impact to our workflows.

Nightwatch itself hasn't made this step yet which is why Night Patrol has to have the workarounds that it does, but hopefully it'll come in time.

In general I hope this trends continues and more users reap the benefits of it.

Always Write Tests

As a prototype, I thought I could get away with not writing tests for the project. But eventually as the project grew and became usable, I realised the lack of tests made changes exponentially harder. I just didn't know if I'd break anything since there's so many paths in the interface.

When I looked into writing tests, I couldn't find anything similar to something like Selenium and WebDriver. Maybe I lack the knowledge to find one or one doesn't exist. Whatever's the case, I'll be looking into it the next time I work on a CLI tool.

Appendix

[1] A typical Nightwatch Configuration

const seleniumServer = require("selenium-server-standalone-jar");
const chromeDriver = require("chromedriver");
const geckoDriver = require("geckodriver");

const launchUrl = "http://localhost:4567";
const windowSize = { width: 1920, height: 1080 };

module.exports = {
  src_folders: "tests/scenarios",
  page_objects_path: "tests/pages",
  custom_commands_path: "tests/commands",
  custom_assertions_path: "tests/assertions",
  globals_path: "tests/globals.js",
  output_folder: ".tmp/tests",

  selenium: {
    start_process: true,
    server_path: seleniumServer.path,
    port: 4444,
    cli_args: {
      "webdriver.chrome.driver": chromeDriver.path,
      "webdriver.gecko.driver": geckoDriver.path,
    },
  },

  test_workers: {
    enabled: true,
    workers: "auto",
  },

  test_settings: {
    chrome: {
      launchUrl,
      desiredCapabilities: {
        browserName: "chrome",
        chromeOptions: {
          args: [
            `--window-size=${windowSize.width},${windowSize.height}`,
            "--mute-audio",
          ],
        },
      },
    },
    firefox: {
      launchUrl,
      desiredCapabilities: {
        browserName: "firefox",
        "moz:firefoxOptions": {
          args: [
            "-width",
            `${windowSize.width}`,
            "-height",
            `${windowSize.height}`,
          ],
        },
      },
    },
  },
};