Test Suites

A Test Suite is a group of reliability tests that get applied to each service in a Gremlin team. Test Suites let you customize how each of your teams evaluates their reliability scores. For example, you can assign unique Test Suites to one or more teams, or use a single Test Suite across your entire organization.

Test Suites are built on Scenarios. Each Scenario appears in the suite as a single test. You can also group Scenarios into categories, such as "Redundancy", "Scalability", and "Depdendencies." Gremlin also provides a default Test Suite called Gremlin Recommended Tests, which uses our built-in Scenarios to provide a balanced collection of test.

Gremlin Recommended Tests

Gremlin provides a default Test Suite called Gremlin Recommended Tests, which uses our built-in Scenarios to provide a balanced collection of test. This suite includes the following tests, organized by category:

• Scalability
  • CPU: Tests that your service scales as expected when CPU capacity is limited. Gremlin will consume CPU in 3 stages (50%, 75%, 90%). Estimated test length: 15 minutes.
  • Memory: Tests that your service scales as expected when memory is limited. Gremlin will increase the memory utilization of your system in 3 stages (50%, 75%, 90%). Estimated test length: 15 minutes.
• Redundancy
  • Host: Tests resilience to host failures by immediately shutting down a randomly selected host or container. Estimated test length: 5 minutes.
  • Zone: Tests your service's availability when a randomly selected zone is unreachable from the other zones. The Gremlin zone tag is required for this test. Estimated test length: 5 minutes.
• Dependencies
  • Failure Test: Drops all network traffic to a specific dependency. Estimated test length: 5 minutes.
  • Latency Test: Delays all network traffic to this dependency by 100ms. Estimated test length: 5 minutes.
  • Certificate Expiry Test: Opens a secure connection to your dependency, retrieves the certificate chain, and validates that no certificates expire in the next 30 days. If there is no secure connection available, and therefore no certificates, this test will pass. Estimated test length: 1 minute.

Creating and editing a Test Suite

Before creating a new Test Suite, you'll first need to create the Scenarios that you want to add to the Test Suite. Follow the instructions in Creating a Scenario if you haven't yet done that.

Once you have your Scenarios created, follow these steps:

• Open the Test Suites page in the Gremlin web app.
• Select + Test Suite.
  • Alternatively, you can click Clone next to an existing suite to create a copy. If you don't want to create an entire suite from scratch, cloning the Gremlin Recommended Tests is a great way to get started!
• Enter a Name for the Test Suite.
• Under Test Suite Composition, use the Search Scenarios box to find the Scenario you want to add to the suite, then click + Add. The Scenario will appear in the list below the search box.
  • You can remove a Scenario from the suite by clicking the Delete button. This won't delete the Scenario; it will simply remove it from the list.
  • Optionally, click the Edit button to change how the Scenario appears in the suite. You can change its name, add a description, and choose a category for the Scenario. Note that changing these settings won't change the Scenario itself, only how the Scenario appears in the Test Suite. You can safely make changes here without affecting the base Scenario.
  • Click Save to save any edits you made to the Scenario.
• Under Teams, use the Search Teams box to find the Gremlin team that you want to assign this Test Suite to. You can add multiple teams at once.
  • Note: each team can only have one Test Suite. Changing a team's Test Suite will also reset the test results and scores to zero, however, the data for the previous Test Suite is preserved and will reappear if you change back to the original Test Suite later.
  • You can remove a team by clicking Remove next to the team name.
• Check the Test Preview section to confirm that this is the correct Scenario, then click Save to save the Test Suite.

Deleting a Test Suite

To delete a Test Suite, open the Test Suites page, find the suite you want to delete, then click the Delete button. Confirm, and the Test Suite will be deleted. This action will not modify any of the Scenarios that were part of the Test Suite.