Private Cloud
- 2 min read
- Last Updated October 17, 2018
GomJabbar
GomJabbar is an open-source implementation of Chaos Monkey written in Java and designed to perform attacks within a private cloud architecture. Attacks are defined through the YAML configuration file and are executed as plain shell commands (e.g. sudo service ${module} stop
). It also integrates with Ansible and Rundeck.
Attacks are defined in the config.yaml
within the commands
block. The fail
value is the fault command to be executed, while the (optional) revert
value is executed to attempt reversion. For example, here the shutdown_service
command calls sudo service <service-name> stop
as a fault and the opposite to revert.
1commands:2 kill_service:3 description: 'Kills services.'4 fail: 'sudo pkill -s 9 -f ${module}'5 revert: 'sudo service ${module} start'67 shutdown_service:8 description: 'Shuts down services.'9 fail: 'sudo service ${module} stop'10 revert: 'sudo service ${module} start'
The filters
block in config.yaml
defines a list of clusters
, modules
, and tags
to either be included or excluded via include
or exclude
, respectively.
1filters:2 clusters:3 include:4 - gremlin-chaos5 exclude:6 - grandmas-cluster78 modules:9 include:10 - nginx11 - redis-server12 exclude:13 - critical1415 tags:16 include:17 - development18 - production19 exclude:20 - test
As we saw in the commands
block above ${module}
can be referenced in fault commands and will automatically be replaced with the relevant service name. Once configured you can start the server by exporting the generated GJ_OPTIONS
and calling the gomjabbar.sh
script.
1export GJ_OPTIONS="-Dcom.outbrain.gomjabbar.configFileUrl=<config file url> ..."2./gomjabbar.sh
You can then use the REST API to trigger attacks and revert faults.
Failure Injection on Your Private Cloud with Gremlin
Gremlin locates weaknesses in your private cloud architecture before they cause problems. Gremlin makes Chaos Engineering simple, safe, and secure, improving your private cloud's stability and resilience. You can begin executing Chaos Experiments in just a few minutes by signing up for an account and installing Gremlin. Gremlin can run experiments on every major type of private cloud infrastructure including Linux, Docker, Kubernetes, OpenStack, VMware, and many more.
Check out these tutorials to learn how to install Gremlin and start Chaos Engineering your private cloud today.
Muxy
Muxy is an open-source tool written in Go that allows you to tamper with network traffic at the transport, TCP, or HTTP protocol layers of your systems. In can simulate real-world network connectivity problems and can also be extended with plugins via the Plugo interface.
Muxy is configured through a YAML file that defines a number of proxy
and middleware
blocks.
Start by installing Muxy with
go get
.bash1go get github.com/mefellows/muxyCreate a YAML file and paste the following test configuration in it.
yaml1# gremlin.yml2proxy:3 - name: http_proxy4 config:5 host: 0.0.0.06 port: 81817 proxy_host: www.gremlin.com8 proxy_port: 443910# Proxy plugins11middleware:12 - name: http_tamperer13 config:14 request:15 host: www.gremlin.com16 port: 4431718 # Message Delay request/response plugin19 - name: delay20 config:21 request_delay: 250022 response_delay: 25002324 # Log in/out messages25 - name: loggerThis configuration creates a
localhost:8181
proxy that targetswww.gremlin.com:443
. It also creates anhttp_tamperer
middleware that catches requests made towww.gremlin.com:443
and adds a2.5-second
delay to both requests and responses. Finally, it outputs Muxy messages to the default terminal logger.Start Muxy with the following command, ensuring that you specify the location of the YAML configuration file.
bash1muxy proxy --config ./gremlin.ymlbash1# OUTPUT22018/09/18 22:47:39.179395 [INFO] Loading plugin http_tamperer32018/09/18 22:47:39.179415 [INFO] Loading plugin delay42018/09/18 22:47:39.179419 [INFO] Loading plugin logger52018/09/18 22:47:39.179451 [INFO] Loading proxy http_proxy62018/09/18 22:47:39.179465 [DEBUG] HTTP Tamperer Setup()72018/09/18 22:47:39.179476 [DEBUG] Delay Symptom - Setup()82018/09/18 22:47:39.179578 [INFO] HTTP proxy listening on http://0.0.0.0:8181Now that Muxy is listening at
http://0.0.0.0:8181
you can make a request towww.gremlin.com
through the proxy to test it out. Muxy should add a delay of approximately 5 seconds to the request.bash1time curl -H"Host: www.gremlin.com" http://localhost:8181/bash1# OUTPUT2real 0m5.070s3user 0m0.000s4sys 0m0.004sNow try the same timed test but without passing through the
localhost:8181
proxy, and you'll see Muxy doesn't catch or muck with this request.bash1time curl -o /dev/null https://www.gremlin.combash1# OUTPUT2real 0m0.257s3user 0m0.036s4sys 0m0.012s
Check out the official Muxy documentation for more information on integrating Muxy into your private cloud Chaos Engineering.