Gremlin is a simple, safe and secure way to use Chaos Engineering to improve system resilience. You can use Gremlin with Docker in a variety of ways. It is possible to attack Docker containers and it is also possible to run Gremlin in a container to create attacks against the host or other containers.
This tutorial will provide a walkthrough of the following:
To run Gremlin in a Docker container, view the guide on How to Install and Use Gremlin in a Docker Container.
Before you begin this tutorial, you’ll need the following:
apt-transport-https
package to be able to install gremlin from our repo via HTTPS.In this step, you’ll install Docker.
Add Docker’s official GPG key:
1curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
Use the following command to set up the stable repository.
1sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
Update the apt package index:
1sudo apt-get update
Make sure you are about to install from the Docker repo instead of the default Ubuntu 16.04 repo:
1apt-cache policy docker-ce
Install the latest version of Docker CE:
1sudo apt-get install docker-ce
Docker should now be installed, the daemon started, and the process enabled to start on boot. Check that it's running:
1sudo systemctl status docker
Make sure you are in the Docker usergroup, replace $USER with your username:
1sudo usermod -aG docker $USER
Log out and back in for your permissions to take effect, or type the following:
1su - ${USER}
htop is an interactive process viewer for unix.
Create the docker file:
1vim Dockerfile
Add the following to the Dockerfile:
1FROM alpine:latest2RUN apk add --update htop && rm -rf /var/cache/apk/*3ENTRYPOINT ["htop"]
Build the Dockerfile and tag the image:
1docker build -t htop .
Run htop inside a container:
1docker run -it --rm --pid=host htop
To exit htop, use the q key.
Next we will create an nginx container and monitor the new container directly by joining the nginx container’s pid namespace.
First we will create a directory for the html page we will serve using nginx:
1mkdir -p ~/docker-nginx/html2cd ~/docker-nginx/html
Create a simple html page:
1vim index.html
Paste in the content shown below:
1<html>2 <head>3 <title>Docker nginx tutorial</title>4 <link5 rel="stylesheet"6 href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css"7 integrity="sha384-Gn5384xqQ1aoWXA+058RXPxPg6fy4IWvTNh0E263XmFcJlSAwiGgFAW/dAiS6JXm"8 crossorigin="anonymous"9 />10 </head>11 <body>12 <div class="container">13 <h1>Hello this is your container speaking.</h1>14 <p>This page was created by your Docker container.</p>15 <p>Now it’s time to create a Gremlin attack.</p>16 </div>17 </body>18</html>
Create a container using the nginx Docker image:
1sudo docker run -l service=nginx --name docker-nginx -p 80:80 -d -v ~/docker-nginx/html:/usr/share/nginx/html nginx
View the nginx Docker container
1sudo docker ps -a
You will see the following:
1CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES2352609a67e95 nginx "nginx -g 'daemon of…" 33 seconds ago Up 32 seconds 0.0.0.0:80->80/tcp docker-nginx
htop can be used to monitor Gremlin attacks against the host and Gremlin attacks against individual containers.
Join the docker-nginx container’s pid namespace:
1docker run -it --rm --pid=container:docker-nginx htop
Before the attack, htop will show you that CPU is not spiking:
11 [| 0.7%] Tasks: 3, 0 thr; 1 running2 2 [|| 1.3%] Load average: 0.07 0.05 0.063 Mem[|||||||||||||||||||||||||| 141M/3.86G] Uptime: 02:48:434 Swp[ 0K/0K]56 PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command7 10 root 20 0 4324 1708 936 R 0.0 0.0 0:00.05 htop8 1 root 20 0 32428 5080 4400 S 0.0 0.1 0:00.03 nginx: master process nginx -g daemon off;9 5 101 20 0 32900 3060 1824 S 0.0 0.1 0:00.00 nginx: worker process
Next we are going to install Gremlin on the host to perform attacks.
First, add the Gremlin Debian repository:
1echo "deb https://deb.gremlin.com/ release non-free" | sudo tee /etc/apt/sources.list.d/gremlin.list
Import the repo’s GPG key:
1sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C81FC2F43A48B25808F9583BDFF170F324D41134 9CDB294B29A5B1E2E00C24C022E8EF3461A50EF6
Then install the Gremlin daemon and CLI:
1sudo apt-get update && sudo apt-get install -y gremlind gremlin
After you have created your Gremlin account (sign up here) you will need to find your Gremlin Daemon credentials. Login to the Gremlin App using your Company name and sign-on credentials. These were emailed to you when you signed up to start using Gremlin.
Navigate to Team Settings and click on your Team.
Store your Gremlin agent credentials as environment variables, for example:
1export GREMLIN_TEAM_ID=3f242793-018a-5ad5-9211-fb958f8dc084
1export GREMLIN_TEAM_SECRET=eac3a31b-4a6f-6778-1bdb813a6fdc
Next run the Gremlin Daemon in a Container.
Use docker run to pull the official Gremlin Docker image and run the Gremlin daemon:
1docker run -d --net=host \2 --cap-add=NET_ADMIN --cap-add=SYS_BOOT --cap-add=SYS_TIME \3 --cap-add=KILL \4 -v $PWD/var/lib/gremlin:/var/lib/gremlin \5 -v $PWD/var/log/gremlin:/var/log/gremlin \6 -v /var/run/docker.sock:/var/run/docker.sock \7 -e GREMLIN_TEAM_ID="$GREMLIN_TEAM_ID" \8 -e GREMLIN_TEAM_SECRET="$GREMLIN_TEAM_SECRET" \9 -e GREMLIN_CLIENT_TAGS="foo=bar" \10 gremlin/gremlin daemon
Use docker ps to see all running Docker containers:
1sudo docker ps
1CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESb281e749ac33 gremlin/gremlin "/entrypoint.sh daem…" 5 seconds ago Up 4 seconds relaxed_heisenberg
Jump into your Gremlin container with an interactive shell (replace b281e749ac33 with the real ID of your Gremlin container):
1sudo docker exec -it b281e749ac33 /bin/bash
From within the container, check out the available attack types:
1gremlin help attack-container
1Usage: gremlin attack-container CONTAINER TYPE [type-specific-options]Type "gremlin help attack-container TYPE" for more details: blackhole # An attack which drops all matching network traffic cpu # An attack which consumes CPU resources io # An attack which consumes IO resources latency # An attack which adds latency to all matching network traffic memory # An attack which consumes memory packet_loss # An attack which introduces packet loss to all matching network traffic shutdown # An attack which forces the target to shutdown dns # An attack which blocks access to DNS servers time_travel # An attack which changes the system time. disk # An attack which consumes disk resources process_killer # An attack which kills the specified process
Example: Creating a CPU Attack from the host against the nginx Docker container using the App
You can use the Gremlin App or the Gremlin API to trigger Gremlin attacks. You can view the available range of Gremlin Attacks in Gremlin Help.
The “Hello World” of Chaos Engineering is the CPU Resource Attack. To create a CPU Resource Attack select “Resource” and then “CPU” in the dropdown menu.
The CPU Resource Attack will consume CPU resources based on the settings you select. The most popular default settings for a CPU Resource Attack are pre-selected, a default attack will utilize 1 core for 60 seconds. Before you can run the Gremlin attack you will need to click either Exact hosts to run the attack on or click the Random attack option.
Click Exact and select the host you created the nginx Docker container on, in this example that is 138.68.226.195.
Container labels will enable you to choose containers on your host to attack.
Click to enable container labels, type in the label details of the container.
For this example, the Nginx Docker container label we created is set to service=nginx.
1sudo docker run -l service=nginx --name docker-nginx -p 80:80 -d -v ~/docker-nginx/html:/usr/share/nginx/html nginx
Finally select ”Create” to kick off a random Gremlin CPU Resource Attack on the nginx Docker container.
Your attack will begin to run, you will be able to view its progress via Gremlin Attacks in the Gremlin Control Panel.
To view the results of the attack join the docker-nginx container’s pid namespace:
1docker run -it --rm --pid=container:docker-nginx htop
You will see the following in htop:
11 [ 0.0%] Tasks: 4, 1 thr; 2 running2 2 [||||||||||||||||||||||||||||||||||||||||||||||||||||100.0%] Load average: 0.61 0.17 0.063 Mem[|||||||||||||||||||||| 176M/3.86G] Uptime: 00:37:174 Swp[ 0K/0K]5 PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command6 18 root 20 0 15456 13692 4112 S 100. 0.3 0:26.39 gremlin attack cpu -c 1 -l 607 13 root 20 0 4324 1988 944 R 0.0 0.0 0:00.12 htop8 1 root 20 0 32428 5184 4504 S 0.0 0.1 0:00.04 nginx: master process nginx -g daemon off;9 8 101 20 0 32900 2948 1712 S 0.0 0.1 0:00.00 nginx: worker process
If you have the Gremlin Slackbot enabled you will also see the bot post that the Gremlin Attack has started. When it’s successful the Gremlin Slackbot will post again. To setup the Gremlin Slackbot follow the guide, How to Setup and Use the Gremlin Slackbot.
When your attack is finished it will move to Completed Attacks in the Gremlin App.
To view the logs of the Attack, click on the Attack in Completed Attacks.
You've installed Gremlin on a server running Docker and validated that Gremlin works by running the “Hello World” of Chaos Engineering for Docker Containers, the CPU Resource attack. You now possess tools that make it possible for you to explore additional Gremlin Attacks.
Gremlin’s Developer Guide is a great resource and reference for using Gremlin to do Chaos Engineering. You can also explore the Gremlin Community for more information on how to use Chaos Engineering with your infrastructure.
Gremlin empowers you to proactively root out failure before it causes downtime. See how you can harness chaos to build resilient systems by requesting a demo of Gremlin.
Get started