OpenShift
- 2 min read
- Last Updated October 17, 2018
Monkey-Ops
Monkey-Ops is an open-source Chaos Monkey implementation written in Go and designed to be deployed alongside an OpenShift application. Monkey-Ops will randomly perform one of two possible attacks:
- Delete a random pod by calling the
DELETE /api/v1/namespaces/{namespace}/pods
Kubernetes API endpoint. - Scale the number of replicas for the associated deployment config by calling the
PUT /oapi/v1/namespaces/{namespace}/deploymentconfigs/{name}/scale
OpenShift API endpoint.
You can install Monkey-Ops either via Docker or as a separate OpenShift project.
Docker Installation
Create a Docker container with the following command. Be sure to replace TOKEN
with your own OpenShift auth token and PROJECT_NAME
with the appropriate value.
1docker run joseangelsilvag/monkey-ops /monkey-ops \2 --TOKEN="<TOKEN>" \3 --PROJECT_NAME="chaos-demo" \4 --API_SERVER="https://api.starter-us-west-2.openshift.com:443" \5 --INTERVAL=30 \6 --MODE="background"
This will randomly execute one of the two possible attacks every INTERVAL
seconds. If you wish to have more control over attacks, change MODE
to "rest"
and use the /chaos
REST API to launch an attack.
OpenShift Installation
Installing Monkey-Ops as an OpenShift project is a bit more complex.
Clone the Git repo to a local directory.
bash1git clone https://github.com/joseangelsilvag/monkey-ops.gitCreate a
monkey-ops.json
file and paste the following, which will be used to create a Service Account.json1{2 "apiVersion": "v1",3 "kind": "ServiceAccount",4 "metadata": {5 "name": "monkey-ops"6 }7}Create the OpenShift Service Account using the OpenShift CLI and grant it privileges for your project (e.g.
chaos-demo
).bash1oc create -f monkey-ops.json && oc policy add-role-to-user edit system:serviceaccount:chaos-demo:monkey-opsNow create a new pod using the
monkey-ops-template.yaml
found in the Monkey-Ops project.bash1oc create -f ./openshift/monkey-ops-template.yaml -n chaos-demoFinally, create a new app called
monkey-ops
and pass appropriate values for eachPARAM
indicating when and how attacks will be executed.bash1oc new-app \2 --name=monkey-ops \3 --template=monkey-ops \4 --param APP_NAME=monkey-ops \5 --param INTERVAL=30 \6 --param MODE=background \7 --param TZ=America/Los_Angeles \8 --labels=app_name=monkey-ops -n chaos-demo
Engineering Chaos In OpenShift with Gremlin
Gremlin simplifies your Chaos Engineering workflow for OpenShift by making it safe and effortless to execute Chaos Experiments across all application containers. As a distributed architecture OpenShift is particularly sensitive to instability and unexpected failures. Gremlin can consume resources, impact network traffic, and perform instance shutdowns on your OpenShift applications.
Check out this tutorial for installing Gremlin on CentOS or this guide for installing Gremlin on OpenShift via a Kubernetes DaemonSet to get started!
Pumba
As discussed in the Chaos Monkey Alternatives - Docker chapter, Pumba is a Chaos injection tool primarily built for Docker. However, it can also be deployed on Kubernetes and, by extension, on OpenShift using a DaemonSet. Pumba can stop, pause, kill, and remove containers, which means it works fairly well with OpenShift pods that are made up of one or more containers.
To deploy Pumba in OpenShift nodes using a DaemonSet you must first add a security policy to allow the OpenShift
developer
user to administer Kubernetes clusters.bash1oc adm policy --as system:admin add-cluster-role-to-user cluster-admin developerAdd the
privileged
security context restraint to thedefault
user for your project.bash1oc adm policy add-scc-to-user privileged system:serviceaccount:<project>:defaultSet the
allowHostDirVolumePlugin
option totrue
in therestricted
security restraint, which will allow OpenShift to connect to the Docker container.bash1oc edit scc restrictedbash1# Please edit the object below. Lines beginning with a '#' will be ignored,2# and an empty file will abort the edit. If an error occurs while saving this file will be3# reopened with the relevant failures.4#5allowHostDirVolumePlugin: true6allowHostIPC: false7allowHostNetwork: false8allowHostPID: false9allowHostPorts: false10allowPrivilegedContainer: false11allowedCapabilities: null12apiVersion: security.openshift.io/v113# [...]Download the pumba_openshift.yml file and modify it as necessary. By default every 30 seconds it will kill a container within a pod containing the string
"hello"
in its name.bash1curl -O https://raw.githubusercontent.com/alexei-led/pumba/master/deploy/pumba_openshift.ymlyaml1apiVersion: extensions/v1beta12kind: DaemonSet3metadata:4 name: pumba5spec:6 template:7 metadata:8 labels:9 app: pumba10 name: pumba11 spec:12 containers:13 - image: gaiaadm/pumba:master14 imagePullPolicy: Always15 name: pumba16 command: ['pumba']17 args:18 [19 '--random',20 '--debug',21 '--interval',22 '30s',23 'kill',24 '--signal',25 'SIGKILL',26 're2:.*hello.*',27 ]28 securityContext:29 runAsUser: 030 volumeMounts:31 - name: dockersocket32 mountPath: /var/run/docker.sock33 volumes:34 - hostPath:35 path: /var/run/docker.sock36 name: dockersocketFinally, create the DaemonSet from the
pumba_openshift.yml
.bash1oc create -f pumba_openshift.yml2daemonset.extensions "pumba" created
That's it. Now just add some pods to your project that match the regex used in the DaemonSet, if any, and Pumba should pick up on them and start killing them off. Check out this handy video tutorial for all the details.