Docker
- 2 min read
- Last Updated October 17, 2018
Pumba
Pumba is a powerful Chaos testing tool for injecting Chaos in Docker. It can kill, pause, stop, and remove Docker containers with highly-configurable selection rules. It can also perform network emulation through delays, packet loss, rate limiting, and more.
Get started by downloading the latest binary release and setting its permissions.
1sudo curl -L https://github.com/alexei-led/pumba/releases/download/0.5.2/pumba_linux_amd64 -o /usr/bin/pumba &&2sudo chmod +x /usr/bin/pumba
gaiaadm/pumba
image. The -v
flag bind-mounts the local docker.sock
file to the same path in the container, so Pumba can connect to the Docker daemon. The --rm
flag makes the container temporary and -it
creates an interactive shell so we can pass the pumba
command that follows.1docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock gaiaadm/pumba \ | <PUMBA_COMMAND> |
Killing a Random Container
Create a Docker container.
bash1docker run -l service=nginx --name nginx -p 80:80 -d nginx2docker container lsbash1# OUTPUT2CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES3b9df13525a13 nginx "nginx -g 'daemon ofโฆ" 4 minutes ago Up 4 minutes 0.0.0.0:80->80/tcp nginxThis command attempts to kill a random container every 30 seconds. The
--dry-run
flag simulates the result, so remove it to perform actual killings.bash1pumba -l info --random --dry-run --interval 30s kill2INFO[0000] killing container app=pumba dryrun=true function=github.com/alexei-led/pumba/pkg/container.dockerClient.KillContainer id=b9df13525a139d9a4a55a249b9cff37ba4656b72b4971fbc1f85d93058f2770d name=/nginx signal=SIGKILL source=container/client.go:115
Network Emulation
Pumba uses the tc
utility for performing network emulation, which is typically installed with the iproute2
tool set. We'll be creating containers using Alpine Linux distributions in these samples, but make sure your own container images contain a copy of the tc
utility when performing network emulations.
Causing Delays
Issue the following command to create a container named
networker
. This ensuresiproute2
is up to date and performs a ping ongoogle.com
for testing.bash1docker run --rm --name networker -it alpine sh -c "apk add --update iproute2 && ping google.com"bash1# OUTPUT2fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz3fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz4(1/6) Installing libelf (0.8.13-r3)5(2/6) Installing libmnl (1.0.4-r0)6(3/6) Installing jansson (2.11-r0)7(4/6) Installing libnftnl-libs (1.1.1-r0)8(5/6) Installing iptables (1.6.2-r0)9(6/6) Installing iproute2 (4.13.0-r0)10Executing iproute2-4.13.0-r0.post-install11Executing busybox-1.28.4-r1.trigger12OK: 8 MiB in 19 packages13PING google.com (172.217.3.174): 56 data bytes1464 bytes from 172.217.3.174: seq=0 ttl=127 time=8.992 ms1564 bytes from 172.217.3.174: seq=1 ttl=127 time=9.965 ms1664 bytes from 172.217.3.174: seq=2 ttl=127 time=10.332 msOpen a second terminal and issue the following command to cause a
5000
milliseconddelay
over a total of15
seconds.bash1pumba -l info netem --duration 15s delay --time 5000 networkerbash1# TERMINAL 2: OUTPUT2INFO[0000] Running netem command '[delay 5000ms 10ms 20.00]' on container 2a4066e2865ed24464fa458982374795d62df11b0368e0886f77fc62cdc47664 for 15s app=pumba function=github.com/alexei-led/pumba/pkg/container.dockerClient.NetemContainer source=container/client.go:2203INFO[0000] start netem for container app=pumba dryrun=false function=github.com/alexei-led/pumba/pkg/container.dockerClient.startNetemContainer id=2a4066e2865ed24464fa458982374795d62df11b0368e0886f77fc62cdc47664 iface=eth0 name=/networker netem=delay 5000ms 10ms 20.00 source=container/client.go:276 tcimage=4INFO[0015] stopping netem on container IPs=[] app=pumba dryrun=false function=github.com/alexei-led/pumba/pkg/container.dockerClient.StopNetemContainer id=2a4066e2865ed24464fa458982374795d62df11b0368e0886f77fc62cdc47664 iface=eth0 name=/networker source=container/client.go:240 tc-image=5INFO[0015] stop netem for container IPs=[] app=pumba dryrun=false function=github.com/alexei-led/pumba/pkg/container.dockerClient.stopNetemContainer id=2a4066e2865ed24464fa458982374795d62df11b0368e0886f77fc62cdc47664 iface=eth0 name=/networker source=container/client.go:298 tcimage=bash1# TERMINAL 1: OUTPUT264 bytes from 172.217.3.174: seq=509 ttl=127 time=9.638 ms364 bytes from 172.217.3.174: seq=512 ttl=127 time=5013.608 ms464 bytes from 172.217.3.174: seq=514 ttl=127 time=5011.516 ms564 bytes from 172.217.3.174: seq=510 ttl=127 time=9299.192 ms664 bytes from 172.217.3.174: seq=511 ttl=127 time=9297.367 ms764 bytes from 172.217.3.174: seq=516 ttl=127 time=5011.184 ms864 bytes from 172.217.3.174: seq=513 ttl=127 time=9301.741 ms964 bytes from 172.217.3.174: seq=518 ttl=127 time=5016.096 ms1064 bytes from 172.217.3.174: seq=519 ttl=127 time=5014.941 ms1164 bytes from 172.217.3.174: seq=515 ttl=127 time=9304.069 ms1264 bytes from 172.217.3.174: seq=527 ttl=127 time=10.468 ms
Dropping Packets
Issue the following command to create a container named
networker
and have it start downloading a fairly large file viacurl
.bash1docker run --rm --name networker -it alpine sh -c "apk add --update iproute2 && apk add --update curl && curl -O http://ubuntu-releases.eecs.wsu.edu/18.04.1/ubuntu-18.04.1-desktop-amd64.iso"bash1# TERMINAL 1: OUTPUT2% Total % Received % Xferd Average Speed Time Time Time Current3 Dload Upload Total Spent Left Speed48 1862M 8 155M 0 0 9698k 0 0:03:16 0:00:16 0:03:00 11.4MOpen a second terminal and issue the
loss
command, which will drop25%
of all packets for the next2
minutes.bash1pumba netem --duration 2m loss --percent 10 networkerYou should notice the packet loss affecting the
curl
download -- in this case, roughly halving download speeds.bash1# TERMINAL 1: OUTPUT2 % Total % Received % Xferd Average Speed Time Time Time Current3 Dload Upload Total Spent Left Speed413 1862M 13 259M 0 0 7403k 0 0:04:17 0:00:35 0:03:42 5807k
Injecting Failure Into Docker with Gremlin
Gremlin makes it easy to run Chaos Experiments on Docker containers. You can start running experiments in just a few minutes after installing Docker. Once installed, Gremlin is intelligent enough to recognize each of your unique Docker containers and will accurately apply smart identifier tags, so you can target exactly the right services and systems. Use Gremlin to perform shutdown and CPU experiments against Docker containers.
Check out this tutorial to learn how to install Gremlin on Ubuntu and attack Docker containers. Alternatively, this guide shows how to install Gremlin within a Docker container for use against other containers.
Docker Chaos Monkey
Docker Chaos Monkey is a simple shell script that terminates Docker Swarm services. Targetable services are specified by applying the role=disposable
label.
1docker service create -l role=disposable --name nginx nginx:stable
The script kills off the first Docker image with the role=disposable
label that also meets the following criteria:
- Must have more than
1
replica. - Actual and desired replica counts must be equivalent.
Here it is in action.
1./chaos.sh
1# OUTPUT2----------------------------------------------------------------------------3| Running this script will kill off 1 docker image with label: role=disposable4| You have 5 seconds to change your mind and CTRL+C out of this.5----------------------------------------------------------------------------6hsn3ezlkqow7 nginx replicated 2/2 nginx:stable7jam29chanegg nginx2 replicated 1/1 nginx:stable8----------------------------------------------------------------------------910hsn3ezlkqow7 nginx: swarm111removing a container12> nginx.2.zecjcxha6zbr0bpfqb017v8vb1314jam29chanegg nginx2: service has only one running container - skipping
Docker Simian Army
The Docker Simian Army is a Docker image of the Simian Army Java toolset. It doesn't provide any additional features on its own, but it's a useful alternative to installing the Simian Army locally. You can test it out in dry mode with the following command.
1docker run -d \2 -e SIMIANARMY_CLIENT_AWS_ACCOUNTKEY=$AWS_ACCESS_KEY_ID \3 -e SIMIANARMY_CLIENT_AWS_SECRETKEY=$AWS_SECRET_ACCESS_KEY \4 -e SIMIANARMY_CLIENT_AWS_REGION=$AWS_REGION \5 -e SIMIANARMY_CALENDAR_ISMONKEYTIME=true \6 -e SIMIANARMY_CHAOS_ASG_ENABLED=true \7 mlafeldt/simianarmy
Add the -d -p 8080:8080
flag to forward port 8080
and connect to the Simian Army REST API.