How to create a custom Test Suite

Andre Newman

Sr. Reliability Specialist

Last Updated: November 16, 2023

Categories: Chaos Engineering, Kubernetes, Reliability Management

Introduction

In this tutorial, we'll show you how to create your own custom suite of Chaos Engineering experiments called a Test Suite. This includes creating a new Test Suite from a set of Scenarios built in to Gremlin, running the Test Suite on a service, and getting a reliability score.

Overview

In Gremlin, you can link one or more Chaos Engineering experiments together in a Scenario. Scenarios are advanced workflows that test your systems against complex failure conditions, like cascading network outages and data center outages. Gremlin also features Test Suites, which let you run multiple Scenarios as a single testing harness. Test Suites are useful for automating your testing process, as well as creating standard sets of Scenarios that you can apply to any team in your Gremlin organization.

For example, imagine we have multiple engineering teams working on deploying applications to Kubernetes. We can take Gremlin's Kubernetes Scenarios, bundle them into a Test Suite, then assign that Test Suite to every team that works in Kubernetes. Whenever those teams add a new service to Gremlin, we can immediately test that service by running the Test Suite. The Test Suite also makes up the service's reliability score, giving us an accurate measure of reliability using standards that we set ourselves. Best of all, we don't need to do any configuration or customization before running the Test Suite on the service.

Prerequisites

Before you begin, make sure you have:

• A Gremlin account (log into an existing account or sign up for a free trial).
  • Note: this tutorial requires an account with the Company Admin or Company Owner roles. See Company Roles for details.
• An application that you can use for testing. This doesn't need to be a Kubernetes application - a simple installation of a web server like Nginx will work.
• The Gremlin agent running on the same host system as your application.
  • To install Gremlin, follow the installation guide in our documentation. Once your host(s) appear in the Gremlin web app, continue to the first step.

Step 1 - Create a new Test Suite

In this step, you’ll create a new Test Suite. For this tutorial, we'll use Gremlin's built-in Recommended Scenarios, but you can choose your own if you prefer.

• First, log into the Gremlin web app and click Test Suites in the left-hand navigation bar. Then, click + Test Suite to create a new suite.
• Give your Test Suite a name that briefly describes the Test Suite. For example, we'll call ours "Kubernetes Test Suite".
• Under Tests, use the Search Scenarios box to find a Scenario to add to the Test Suite. You can scroll through the list, or start typing to filter the list. In this example, search for "Validate Auto-Scaling with Health Checks" and add the result listed under "Recommended & Shared". If you have a different Scenario you'd prefer to use, you can search for it instead.
  • This list includes user-created Scenarios and Scenarios built in to Gremlin (i.e. Recommended Scenarios). If you're getting many results, you can use the Filter Scenarios drop-down box to reduce the number of Scenarios shown.
  • When you've chosen a Scenario, click + Add to add it to your Test Suite. Here, you can choose which category the Scenario should go in. Categories group one or more Scenarios within the Test Suite. If you're using the auto-scaling Scenario, we recommend choosing "Scalability".
  • Click Save to save the Scenario to your suite.
• If you want to add more Scenarios, repeat step 2 until you've added all of the ones you want.
• Next, choose which Teams will adopt the Test Suite. These correspond to teams within your Gremlin organization. Adding a team and saving the Test Suite immediately changes each of those teams to use the new Test Suite. Data from previous Test Suites (runs, results, etc.) will be retained, so you can switch back at any time without losing scores or test history. At the very least, choose the team that you're a member of so you can run the Test Suite on your own services.
  • Note: You can choose not to add any teams now, then add them later.
• Click Save to save the Test Suite. The new suite will appear in the list.

Step 2 - Run a test

Next, let's run our new Test Suite on a service.

• Click on Services in the left-hand navigation bar to open the Services page.
• Find a service that you want to test. If you haven't yet added any services, follow the instructions in the quick-start guide or documentation. Any application will work for this example (web servers like Nginx or Apache are recommended because they make it easy to directly observe the impact of a running experiment). Once you've found a service, click on its name.
• When viewing the service's overview page, scroll down to Reliability Tests and look for the "Validate Auto-Scaling with Health Checks" test. Click Run to start it.

Gremlin will now run the test and monitor your service using its Health Check(s). If the test completes without your Health Check reporting a single failure, the test passes. If the Health Check fails or times out, then the test is marked as failed and you'll only get partial points.

And that's it! You now have a custom Test Suite that you can assign to any team in your Gremlin organization!

Next steps

This tutorial only covers the basics of creating a Test Suite. From here, you can expand your Test Suite by adding Scenarios, assigning the suite to more teams, or by cloning your Test Suite and editing a copy. You can also create and add custom Scenarios to your Test Suite and run them alongside (or in place of) our Recommended Scenarios. You can have as many Test Suites as you like, and you can even have a different Test Suite for each team. Just remember that each team can only be assigned one Test Suite.

If you'd like to learn more about Test Suites, check out our documentation.