Additional Configuration for Helm
Some environments require additional configuration. Review the following sections to find the best configuration for your environment and then verify your installation.
- Declare container driver
- Enable Gremlin on the Kubernetes Master
- Add AppArmor support
- Use a PodSecurityPolicy
- Use a custom seccomp policy
- Configure a proxy
- Share namespaces with other Gremlin teams
Declare container driver
Gremlin currently has 4 different drivers for integrating with the underlying container runtime powering Kubernetes, as shown in the following table. Gremlin automatically chooses any of the cgroup drivers when the associated requirements are met.
When using Helm, you can declare the intended container driver with the following:
11--set gremlin.container.driver=$driver
Driver | Requirements and file access | Notes |
---|---|---|
containerd-runc | Connect: /run/containerd/containerd.sock | Used with containerd container runtime |
Write: /run/containerd/runc/k8s.io | ||
Write: host's cgroup root | ||
Access to the host's PID namespace | ||
crio-runc | Connect: /run/crio/crio.sock | Used with Cri-O container runtime |
Write: /run/runc | ||
Write: host's cgroup root | ||
Access to the host's PID namespace | ||
docker | Connect: /var/run/docker.sock | No support for systemd cgroup driver |
docker-runc | Connect: /var/run/docker.sock | Recommended for Docker runtime |
Write: /run/docker/runtime-runc/moby | ||
Write: host's cgroup root | ||
Minimum Docker version: 17.11.0 |
Enable Gremlin on the Kubernetes Master
Most Kubernetes deployments configure master nodes with the node-role.kubernetes.io/master:NoSchedule
taint. You can run the following command to see if any of your nodes have this taint:
1kubectl get no -o=custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
1NAME TAINTS2kube-01 [map[effect:NoSchedule key:node-role.kubernetes.io/master]]3kube-02 <none>
To install Gremlin on a Kubernetes master that has been tainted, add those tolerations to your Helm arguments:
1--set tolerations\[0\].effect=NoSchedule \2--set tolerations\[0\].key=node-role.kubernetes.io/master \3--set tolerations\[0\].operator=Exists
Add AppArmor support
If your cluster has AppArmor enabled (for example, Azure Kubernetes Service), add the following line to your Helm deployment to allow the Gremlin container to run without a security profile:
1--set gremlin.apparmor=unconfined
Use a PodSecurityPolicy
Gremlin does not support running within the restricted
PodSecurityPolicy (PSP) that is configured by default on clusters that enable such policies. You can install a gremlin
PodSecurityPolicy to grant chao
and gremlin
everything they need, and nothing they don't need.
When installing Gremlin with Helm, you can supply --set gremlin.podSecurity.podSecurityPolicy.create=true
to install Gremlin's custom pod security policies. Check out Gremlin's Helm Chart Repository for full documentation and usage.
Use a custom seccomp policy
All Gremlin behavior works under Docker's default seccomp policy. However some environments use seccomp profiles that are more restrictive, and prevent Gremlin behavior when using their default seccomp profile.
Gremlin has a custom seccomp profile which can be automatically installed when you install with Helm and pass --set gremlin.podSecurity.seccomp.enabled=true
. Check out Gremlin's Helm Chart Repository for full documentation and usage.
Configure a proxy
Both Gremlin and Chao can be configured to use a proxy for outgoing HTTP traffic. The conventional https_proxy
and no_proxy
variables can be passed as environment variables for this purpose.
When installing Gremlin with Helm, the proxy configuration is automated. See examples.
When experiments are done on Kubernetes resources, the proxy settings are carried along. Ensure your configuration will allow the gremlin sidecar to continue talking with the Attack Console for the duration of the experiment while it is attached to the target resource.
Proxy certificate authorities
When proxies support HTTPS communication, or are otherwise configured with a TLS certificate, it can be necessary to configure Gremlin to trust the proxy's certificate authority. This is done by passing the SSL_CERT_FILE
environment variable where the value is a path on the file system to a PEM encoded file containing the certificate authority's certificate.
Configuring Gremlin
1- name: https_proxy2 value: http://proxy.local:31283# Pass SSL_CERT_FILE when the proxy requires trusting a TLS certificate4- name: SSL_CERT_FILE5 value: /etc/gremlin/ssl/proxy-ca.pem
Configuring the Gremlin Kubernetes Agent
Because the Gremlin Kubernetes Agent (Chao) communicates with the local Kubernetes ApiServer in addition to the internet, it's important to bypass internet proxies for traffic bound to apiserver
1- name: https_proxy2 value: http://proxy.local:31283- name: no_proxy4 value: $(KUBERNETES_SERVICE_HOST):$(KUBERNETES_SERVICE_PORT)5# Pass SSL_CERT_FILE when the proxy requires trusting a TLS certificate6- name: SSL_CERT_FILE7 value: /etc/gremlin/ssl/proxy-ca.pem8# Pass SSL_CERT_DIR when SSL_CERT_FILE contains only the proxy certificate. This will ensure Chao trusts api.gremlin.com9# The value of SSL_CERT_DIR varies depending on the operating system on which the cluster hosts run10# See /docs/infrastructure-layer/kubernetes/#ssl_cert_dir11- name: SSL_CERT_DIR12 value: /etc/ssl/
SSL_CERT_DIR
Supplying SSL_CERT_DIR
ensures Chao is still configured with the necessary certificate authories to trust api.gremlin.com
. However it is not needed for most Gremlin installations because Chao will trust Gremlin servers by default. This variable is only required for Chao deployments when both of the following conditions are true:
- Chao is configured with
https_proxy
and this proxy is configured to accept TLS connections - Chao is also configured with
SSL_CERT_FILE
, and the file it points to contains only the certificate authority for the https proxy
The value of SSL_CERT_DIR
should point to the root of the certificate authority directory for the operating system on which Chao runs.
Path | OS |
---|---|
/etc/ssl/certs/ | Debian/Ubuntu |
/etc/pki/tls/ | Fedora/RHEL 6/OpenELEC |
/etc/ssl/ | OpenSUSE / Alpine Linux |
/etc/pki/ca-trust/extracted/pem/ | CentOS/RHEL 7 |
Share namespaces
With the Gremlin Kubernetes Agent installed on your cluster you can share individual namespaces with other Gremlin teams. Once installed head to the Agents list in the Gremlin web app to view all of the clusters installed across your company.
By sharing individual namespaces to teams across your company, you can provide access for users to run experiments only on relevant services while also limiting access to the hosts or nodes themselves.
An updated Gremlin Kubernetes Agent is required, no earlier than July 6th, 2020, to manage access control on your cluster.
Managing Cluster access
As the Team Manager
on a team where a Kubernetes cluster is installed or as a Company Manager
, you can click the gear icon to manage access. On the cluster view, to share a namespace
with a team use the search box to filter down the list of available teams
. Then use the search box on the team row and click on the namespace
you'd like to share. Use the options menu to share all of the namespaces.
To remove access of a namespace to a team, click on the x
on the blue namesapce bubbles. Using the options menu you can also remove all namespaces at once.
Requesting Namespace access
As a member of a different team of your company, you can view the list of clusters installed across your company. To request access to a namespace on one of these clusters not installed on your team, click the Request Access
button. You can then check off the namespaces you'd like access to, or you can use the select all switch.
You can also request access to a namespace within a cluster when creating an experiment. Once you've selected a cluster, the drop down list of namespaces will have an option to request access.
Approving access requests
To approve or deny an access request, you must be a Team Manager
. Navigate to the Agents list, locate your cluster, and click the gear icon to the right. On the Manage Cluster Access page, you'll see any pending requests from your company.
Adding annotations to service account
You can use the advanced feature of adding custom annotations to the gremlin and chao service accounts in Kubernetes. For example, this can be used to tag the gremlin or chao service accounts with an AWS EKS IAM role when running:
1helm install gremlin gremlin/gremlin \2 --namespace gremlin \3 --set gremlin.serviceAccount.annotations."eks\.amazonaws\.com\/role-arn"="arn:aws:iam::123412341234:role/K8sServiceAccountRole" \4 --set chao.serviceAccount.annotations."eks\.amazonaws\.com\/role-arn"="arn:aws:iam::123412341234:role/ChaoK8sServiceAccountRole"
Adding environment variables
You can use the advanced feature of adding custom environment variables to gremlin and chao services.
1helm install gremlin gremlin/gremlin \2 --namespace gremlin \3 --set 'gremlin.extraEnv[0].name=TEST1' \4 --set 'gremlin.extraEnv[0].value=hello'
Sets environment variable TEST1 to hello