Kubernetes
- 2 min read
- Last Updated October 17, 2018
Kube Monkey
Kube-monkey is an open-source implementation of Chaos Monkey for use on Kubernetes clusters and written in Go. Like the original Chaos Monkey, Kube-monkey performs just one task: it randomly deletes Kubernetes pods within the cluster, as a means of injecting failure in the system and testing the stability of the remaining pods. It is based on pseudo-random rules, running at a pre-defined hour on weekdays to then build a schedule. Based on the generated schedule random pod targets that will be attacked and killed at a random time during that same day, although the time-range is configurable.
Kube-monkey will only terminate pods that have explicitly opted in by specifying certain Kube-monkey metadata labels
. The following illustrates the basic labels that can be specified to allow Kube-monkey to kill pods within the application.
1apiVersion: extensions/v1beta12kind: Deployment3metadata:4 name: monkey-victim5 namespace: app-namespace6 labels:7 kube-monkey/enabled: enabled8 kube-monkey/identifier: monkey-victim9 kube-monkey/mtbf: '2'10 kube-monkey/kill-mode: 'fixed'11 kube-monkey/kill-value: 112spec:13 template:14 metadata:15 labels:16 kube-monkey/enabled: enabled17 kube-monkey/identifier: monkey-victim18# ...
Check out the GitHub repository for more information on installing and using Kube-monkey.
Engineering Chaos In Kubernetes with Gremlin
Gremlin simplifies your Chaos Engineering workflow for Kubernetes by making it safe and effortless to execute Chaos Experiments across all nodes. As a distributed architecture Kubernetes is particularly sensitive to instability and unexpected failures. Gremlin can overload resources, impact network traffic, shutdown nodes, and more on your Kubernetes clusters.
Check out this tutorial over on our community site to get started!
Kubernetes Pod Chaos Monkey
Kubernetes Pod Chaos Monkey is a Chaos Monkey-style tool for Kubernetes. The code itself is a local shell script that issues kubectl commands to occasionally locate and then delete Kubernetes pods. It targets a cluster based on the configurable NAMESPACE
and attempts to destroy a node every DELAY
seconds (defaulting to 30).
Since Kubernetes Pod Chaos Monkey is essentially a simple shell script it can be modified quite easily.
The Chaos Toolkit
The Chaos Toolkit is an open-source and extensible tool that is written in Python. It uses platform-specific drivers to connect to your Kubernetes cluster and execute Chaos Experiments. Every experiment performed by Chaos Toolkit is written in JSON using a robust API. Experiments are made up of a few key elements that are executed sequentially and allow the experiment to bail out if any step in the process fails.
Steady State Hypothesis: This element defines the normal or "steady" state of the system before the Method element is applied. Here we've defined a basic application with a steady state hypothesis titled "Service should have nodes."
json1{2 "version": "1.0.0",3 "title": "Gremlin EKS App",4 "description": "Gremlin EKS App",5 "tags": ["service", "kubernetes"],6 "steady-state-hypothesis": {7 "title": "Service should have nodes.",8 "probes": [9 {10 "type": "probe",11 "name": "nodes_found",12 "tolerance": true,13 "provider": {14 "type": "python",15 "module": "chaosk8s.node.probes",16 "func": "get_nodes",17 "arguments": {18 "label_selector": "eks-gremlin-chaos"19 }20 }21 }22 ]23 }24}Probe: A Probe is an element that collects system information, such as checking the health status of a node. Here we define a Probe element, which we've added to our steady state Probes list above, that calls the
get_nodes
function and retrieves the list of nodes for the specifiedlabel-selector
.json1{2 "type": "probe",3 "name": "nodes_found",4 "tolerance": true,5 "provider": {6 "type": "python",7 "module": "chaosk8s.node.probes",8 "func": "get_nodes",9 "arguments": {10 "label_selector": "eks-gremlin-chaos"11 }12 }13}Action: An Action element performs an operation against the system, such as draining or deleting a node. In the example we call the
delete_nodes
function, passing the requiredlabel-selector
argument, and settingall
totrue
so we delete all nodes in the cluster.json1{2 "type": "action",3 "name": "delete_all_nodes",4 "provider": {5 "type": "python",6 "module": "chaosk8s.node.actions",7 "func": "delete_nodes",8 "arguments": {9 "all": true,10 "label-selector": "eks-gremlin-chaos"11 }12 }13}Method: A Method element defines the series of Probe and Action elements that make up the experiment. Here we're first using the
nodes_found
Probe to make sure nodes exist, executing thedelete_all_nodes
Action to delete all nodes in the cluster, then performing another explicit Probe to verify that no nodes remain.json1"method": [2 {3 "ref": "nodes_found"4 },5 {6 "type": "action",7 "name": "delete_all_nodes",8 "provider": {9 "type": "python",10 "module": "chaosk8s.node.actions",11 "func": "delete_nodes",12 "arguments": {13 "all": true,14 "label-selector": "eks-gremlin-chaos"15 }16 }17 },18 {19 "type": "probe",20 "name": "nodes_not_found",21 "tolerance": false,22 "provider": {23 "type": "python",24 "module": "chaosk8s.node.probes",25 "func": "get_nodes",26 "arguments": {27 "label_selector": "eks-gremlin-chaos"28 }29 }30 }31]
That's the basics to begin experimenting using the Chaos Toolkit. Chaos Toolkit also has a fault injection plugin for Gremlin so you can easily perform attacks while utilizing the safety and security of the Gremlin platform.