Gremlin is a simple, safe and secure service for performing Chaos Engineering experiments through a SaaS-based platform.
This tutorial will show you how to use Gremlin Scenarios to reproduce the AWS S3 Outage.
If you are interested in learning more about the outage, we have shared a detailed analysis of the 2017 S3 Outage on our Gremlin Blog. A good question to ask yourself at every postmortem is “how do we ensure this never happens again?”
Connect to your host with ssh and create a Dockerfile:
1ssh username@your_server_ip23vim Dockerfile45FROM nginx:alpine6COPY index.html /usr/share/nginx/html/index.html
Create the following index.html file:
1vim index.html
1<html lang="en">2 <head>3 <title>Mythical Mysfits</title>4 <meta charset="utf-8" />5 <meta name="viewport" content="width=device-width, initial-scale=1" />6 <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>7 <script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.5.6/angular.min.js"></script>8 <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.3/umd/popper.min.js"></script>9 <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.1.1/js/bootstrap.min.js"></script>10 <link11 rel="stylesheet"12 href="https://stackpath.bootstrapcdn.com/bootstrap/4.1.1/css/bootstrap.min.css"13 />14 </head>1516 <body ng-app="mysfitsApp">17 <style>18 @media (max-width: 800px) {19 img {20 max-width: 300px;21 }22 }23 </style>2425 <div style="text-align: center;">26 <img27 src="https://www.mythicalmysfits.com/images/aws_mythical_banner.png"28 width="800px"29 align="center"30 />31 </div>3233 <div class="container" ng-controller="mysfitsFilterController">34 <div id="filterMenu">35 <ul class="nav nav-pills">36 <li37 class="nav-item dropdown"38 ng-repeat="filterCategory in filterOptionsList.categories"39 >40 <a41 class="nav-link dropdown-toggle"42 data-toggle="dropdown"43 href="#!"44 role="button"45 aria-haspopup="true"46 aria-expanded="false"47 >48 {{filterCategory.title}}49 </a>5051 <div class="dropdown-menu">52 <button53 class="dropdown-item"54 ng-repeat="filterCategorySelection in filterCategory.selections"55 ng-click="queryMysfits(filterCategory.title, filterCategorySelection)"56 >57 {{filterCategorySelection}}58 </button>59 </div>60 </li>6162 <li class="nav-item ">63 <button64 type="button"65 class="btn btn-success"66 ng-click="removeFilter()"67 >68 View All69 </button>70 </li>71 </ul>72 </div>73 </div>7475 <br />7677 <div class="container">78 <div id="mysfitsGrid" class="row" ng-controller="mysfitsListController">79 <div80 class="col-md-4 border border-warning"81 ng-repeat="mysfit in mysfits"82 >83 <br />8485 <p align="center">86 <strong> {{mysfit.name}}</strong>8788 <br />8990 <img src="{{mysfit.thumbImageUri}}" alt="{{mysfit.Name}}" />91 </p>9293 <p>94 <br />95 <b>Species:</b> {{mysfit.species}}96 <br />97 <b>Age:</b> {{mysfit.age}}98 <br />99 <b>Good/Evil:</b> {{mysfit.goodevil}}100 <br />101 <b>Lawful/Chaotic:</b> {{mysfit.lawchaos}}102 </p>103 </div>104 </div>105 </div>106107 <p>108 <br />109 <br />110 This site was created for use in the AWS Modern Application Workshop.111 <a href="https://github.com/aws-samples/aws-modern-application-workshop"112 >Please see details here.</a113 >114 </p>115 </body>116117 <script>118 var mysfitsApiEndpoint =119 'http://mysfits-nlb-9c8e61c17ef3cd1d.elb.us-east-1.amazonaws.com';120 var app = angular.module('mysfitsApp', []);121 var gridScope;122 var filterScope;123124 app.controller('clearFilterController', function($scope) {});125126 app.controller('mysfitsFilterController', function($scope) {127 filterScope = $scope;128129 // The possible options for Mysfits to populate the dropdown filters.130 $scope.filterOptionsList = {131 categories: [132 {133 title: 'Good/Evil',134 selections: ['Good', 'Neutral', 'Evil'],135 },136 {137 title: 'Lawful/Chaotic',138 selections: ['Lawful', 'Neutral', 'Chaotic'],139 },140 ],141 };142143 $scope.removeFilter = function() {144 allMysfits = getAllMysfits(applyGridScope);145 };146147 $scope.queryMysfits = function(filterCategory, filterValue) {148 var filterCategoryQS = '';149150 if (filterCategory === 'Good/Evil') {151 filterCategoryQS = 'GoodEvil';152 } else {153 filterCategoryQS = 'LawChaos';154 }155156 var mysfitsApi =157 mysfitsApiEndpoint +158 '/mysfits?' +159 'filter=' +160 filterCategoryQS +161 '&value=' +162 filterValue;163164 $.ajax({165 url: mysfitsApi,166167 type: 'GET',168169 success: function(response) {170 applyGridScope(response.mysfits);171 },172173 error: function(response) {174 console.log('could not retrieve mysfits list.');175 },176 });177 };178 });179180 app.controller('mysfitsListController', function($scope) {181 gridScope = $scope;182183 getAllMysfits(applyGridScope);184 });185186 function applyGridScope(mysfitsList) {187 gridScope.mysfits = mysfitsList;188189 gridScope.$apply();190 }191192 function getAllMysfits(callback) {193 var mysfitsApi = mysfitsApiEndpoint + '/mysfits';194195 $.ajax({196 url: mysfitsApi,197198 type: 'GET',199200 success: function(response) {201 callback(response.mysfits);202 },203204 error: function(response) {205 console.log('could not retrieve mysfits list.');206 },207 });208 }209 </script>210</html>
Save the index.html file.
Next, build the Dockerfile by running the following:
1docker build -t simple-nginx .
Now we can run our image by using
1docker run -d -p 8080:80 simple-nginx
Now you can see your sample running @ your_server_ip:8080
The Gremlin Unavailable Dependencies Scenario uses a Blackhole attack. We will be using this Blackhole attack to disallow images stored in S3 from loading. To see the results of this Blackhole Network attack we will be using a service called VNC.
On your host, install the Xfce and TightVNC packages:
1sudo apt-get update23sudo apt install xfce4 xfce4-goodies tightvncserver
To complete the VNC installation run the following command, you will be prompted to enter a password:
1vncserver
Next, test the VNC connection on your local computer. Run the following command which uses port forwarding :
1ssh -L 5901:127.0.0.1:5901 -N -f -l username server_ip_address
Now you can use a VNC client to connect to the VNC server at localhost:5901
. You’ll be prompted to authenticate. Use the password you set up earlier. You can use the built-in program for Mac called Screen Sharing or VNC Viewer to view your Xfce Desktop.
On your host you will need to ensure you have a browser installed, install Firefox by running the following command:
1apt-get install firefox
Before you move onto the next step, ensure that you are able to view the sample app using VNC. Connect to localhost:5901
and click on Applications > Internet > Firefox Web Browser
:
Using Firefox, navigate to localhost:8080
, you should see the following:
Now we’re ready to run the Gremlin Unavailable Dependency Scenario to reproduce the S3 Outage.
First, navigate to Recommended Scenarios within the Gremlin UI and choose the Unavailable Dependency Scenario:
Next, select Add Targets and run. Then select your host using the local-hostname option:
Then click customize:
Next, you will click Add Attacks, this will take you to the Gremlin Attack configuration. To reproduce the S3 outage modify the default scenario to include 1 x Blackhole attack impacting AWS S3 us-east-1. You will need to make these changes in the Providers section of the Blackhole attack, see the screenshot below:
Next, click Unleash Scenario. Your Gremlin Scenario will now be running and it will begin to reproduce the S3 Outage:
To view the S3 Outage being reproduced open your VNC viewer and reload your Firefox tab, you will notice that the images stored in us-east-1 on S3 no longer load:
This tutorial has demonstrated how you can use the Gremlin Recommended Scenario “Unavailable Dependency” to reproduce the AWS S3 Outage. This demonstrates that our sample app is not resilient to this outage.
To ensure we could reliably handle this scenario, we could run this Gremlin Scenario again after rectifying this situation with one of many options:
Gremlin empowers you to proactively root out failure before it causes downtime. See how you can harness chaos to build resilient systems by requesting a demo of Gremlin.
Get started