Search documentation
Dashboard
Getting Started

Configuring the Gremlin Agent

This documentation page shows you how to configure the Gremlin Agent. You can configure Gremlin using either environment variables or a configuration file, although we recommend using the configuration file in most cases. This guide assumes you've already installed and authenticated the Agent.

Using the configuration file

Gremlin's configuration file is located in a file named config.yaml. On Linux, this file lives at /etc/gremlin/config.yaml, and on Windows, its path is C:\ProgramData\Gremlin\Agent\config.yaml. By default, Gremlin ships with a commented-out version of this file. Use the comments or the following example to set up your configuration.

Accepted variables

VariableDescription
identifiera string that uniquely identifies a Gremlin installation (defaults to the hosts IP address)
team_idthe ID of your Gremlin team
tagskey-value pairs that help you target this machine during experiments
team_secretthe secret value (used by secret authentication only)
team_certificatethe public portion of the certificate-pair (required for signature auth only)
team_private_keythe private portion of the certificate-pair (required for signature auth only)
https_proxythe url of a proxy in between Gremlin and api.gremlin.com
ssl_cert_filethe ssl certificate for your https proxy
push_metricsset to false if you would like to disable system metrics used for experiment charting (defaults to true)
collect_processesenables/disables Process Collection (as of Linux Agent 2.25.0 default is true)
iam_rolerole the agent uses to retrieve remote cloud config values

Example: config.yaml

yaml
1## Gremlin Identifier; uniquely identifies this machine with Gremlin
2## AWS ARNs are also supported (see below section on supported services and credentials)
3## (can also set with GREMLIN_IDENTIFIER environment variable)
4identifier: gremlin-01
5
6## Gremlin Team Id; you can find this value at https://app.gremlin.com/settings/teams
7## AWS ARNs are also supported (see below section on supported services and credentials)
8## (can also be set with GREMLIN_TEAM_ID environment variable)
9team_id: 11111111-1111-1111-1111-111111111111
10
11## Gremlin Client Tags; Tag your machine with key-value pairs that help you target this machine during experiments
12## (can also set with GREMLIN_CLIENT_TAGS environment variable)
13tags:
14 service: pet-store
15 interface: http
16
17## Gremlin Team Secret, should not be set when using `team_certificate`+`team_private_key`
18## AWS ARNs are also supported (see below section on supported services and credentials)
19## (can also set with GREMLIN_TEAM_SECRET environment variable)
20#team_secret: 11111111-1111-1111-1111-111111111111
21
22## Gremlin Team Certificate, should not be set when using `team_secret`.
23## Paste certificate content here or a paths to the file (prefixed with `file://`)
24## AWS ARNs are also supported (see below section on supported services and credentials)
25## (can also set with GREMLIN_TEAM_CERTIFICATE_OR_FILE environment variable)
26team_certificate: |
27 -----BEGIN CERTIFICATE-----
28 1111111111111111111111111111111111111111111111111111111111111111
29 1111111111111111111111111111111111111111111111111111111111111111
30 1111111111111111111111111111111111111111111111111111111111111111
31 1111111111111111111111111111111111111111111111111111111111111111
32 1111111111111111111111111111111111111111111111111111111111111111
33 1111111111111111111111111111111111111111111111111111111111111111
34 1111111111111111111111111111111111111111111111111111111111111111
35 1111111111111111111111111111111111111111111111111111111111111111
36 1111111111111111111111111111111111111111111111111111111111111111
37 11111111111111111111111111111111111111111111111111==
38 -----END CERTIFICATE-----
39
40## Gremlin Team Certificate, should not be set when using `team_secret`.
41## Paste certificate content here or a paths to the file (prefixed with `file://`)
42## AWS ARNs are also supported (see below section on supported services and credentials)
43## (can also set with GREMLIN_TEAM_PRIVATE_KEY_OR_FILE environment variable)
44team_private_key: file:///var/lib/gremlin/key.pem
45
46## HTTPS Proxy, set this when routing outbound Gremlin HTTPS traffic through a proxy
47## (can also set with HTTPS_PROXY or https_proxy environment variables)
48https_proxy: https://localhost:3128
49
50## SSL CERT FILE, set this when using a https proxy with a self-signed certificate
51## Paste certificate content here or a paths to the file (prefixed with `file://`)
52## (can also set with SSL_CERT_FILE environment variable)
53ssl_cert_file: file:///var/lib/gremlin/proxy_cert.pem
54
55## Push Metrics, tell Gremlin whether to send system metrics to the control plane for charting the impact of experiments in
56## real time. Metrics are only collected during active experiments, and only metrics relevant to the experiment are collected.
57## defaults to `true`
58## (can also set with PUSH_METRICS environment variable)
59push_metrics: true
60
61## Collect Process Data, data about running processes is sent to Gremlin for service discovery.
62## defaults to `true` as of Gremlin Linux Agent 2.25.0
63collect_processes: true

Restarting the Gremlin Agent

To restart the Gremlin Agent and apply your new changes, run the following commands (depending on your environment):

Linux
shell
1sudo systemctl restart gremlind
Windows
PowerShell
1Restart-Service -Name gremlind

Using environment variables

The daemon supports the following environment variables:

Environment VariableDescription
GREMLIN_TEAM_IDYour Team ID (required for authentication)
GREMLIN_TEAM_SECRETYour Team Secret (should only require secret or PEM certificates, not both)
GREMLIN_TEAM_PRIVATE_KEY_OR_FILEYour PEM-encoded private key or path to a file containing the private key (required for authentication). Paths should be prefixed with file:// . For example, on Unix: file:///var/lib/gremlin/key.pem , on Windows: file://C:/ProgramData/Gremlin/Agent/key.pem
GREMLIN_TEAM_CERTIFICATE_OR_FILEThe PEM-encoded public-key certificate or path/filename to the file containing your PEM-encoded public-key certificate (required for authentication). Paths should be prefixed with file://. For example, on Unix: file:///var/lib/gremlin/cert.pem , on Windows: file://C:/ProgramData/Gremlin/Agent/cert.pem
GREMLIN_IDENTIFIERCustom name for this client (default as the host's IP address)
GREMLIN_CLIENT_TAGSComma-separated list of custom tags to assign to this client. For example: GREMLIN_CLIENT_TAGS="zone=us-east1,role=mysql,foo=bar"

The following are optional environment variables that may be required in Docker containerized deployments:

Environment VariableDescription
GREMLIN_BYPASS_USERNS_REMAPIndicates to run the Gremlin Daemon on the same Docker namespace as the host. By default, Gremlin does not launch with a namespace defined. To set this variable, change the value to 1
GREMLIN_DOCKER_IMAGEIndicates the location and version of the Gremlin Docker image to use. This variable can be used to indicate an internal repository. For example: GREMLIN_DOCKER_IMAGE=registry.hub.docker.com/gremlin/gremlin:latest
GREMLIN_IAM_ROLEIndicates to run the Gremlin Daemon with the specified role when retrieving remote config values
net=hostIndicates to run the Gremlin Daemon to use the host network. By default, Gremlin network experiments will not affect the host network for containerized environments.
pid=hostIndicates to run the Gremlin Daemon to affect host processes. By default, Gremlin process killer will not affect host processes for containerized environments.

In addition, the following standard Linux environment variables allow proxy configuration:

Environment VariableDescription
https_proxyIn the form http[s]://[username:password@]address:port
SSL_CERT_FILEIndicates location of the proxy certificates used to authenticate traffic with the proxy. For Example: SSL_CERT_FILE=/etc/ssl/certs/ca-example.crt

Configuration value priorities

If both the environment variables and the configuration file are used when configuring Gremlin, the environment variable values will take priority. For example if you have the GREMLIN_IDENTIFIER environment variable set to foo and in the config.yaml file the identifier field is set to bar, Gremlin will end up using foo as its identifier when registering with the API.

Remote config values

The Gremlin agent supports using AWS ARNs for the following config values in either ENVVARs or config file:

  • identifier
  • team_id
  • team_secret
  • team_certificate
  • team_private_key

When remote configuration values are used, Gremlin does not store the corresponding configuration values on the hard disk of the system where it runs. When Gremlin is configured to use Secret Authentication option (i.e. using a team_secret), the agent must store temporary session tokens in the .credentials file, which is only readable by the Gremlin system user.

The following services are supported for reading values:

  • ssm
  • secretsmanager

The agent requires some credentials to exist on the machine to be able to retrieve the remote values. The credentials preference follows the default AWS order.

Passing environment variables to Systemd or SysV

While configuring the daemon with config.yaml is recommended, process managers like systemd and sysvinit can pass environment variables directly to the daemon process from /etc/default/gremlind. Your installation comes with an example of this file at /etc/default/gremlind.example

shell
1# Gremlin Identifier; uniquely identifies this machine with Gremlin
2GREMLIN_IDENTIFIER=gremlin-01
3
4# Gremlin Team ID; you can find this value at https://app.gremlin.com/settings/teams
5GREMLIN_TEAM_ID=11111111-1111-1111-1111-111111111111
6
7# `gremlin init` will run automatically when the gremlind service starts.
8# Supply extra options to `gremlin init` via this variable
9GREMLIN_INIT_OPTS="--tag service=pet-store --tag interface=http"
10
11# Gremlin Team Secret, should not be set when using `team_certificate`+`team_private_key`
12GREMLIN_TEAM_SECRET=11111111-1111-1111-1111-111111111111

You can set any of the environment variables listed in the previous section in the configuration file.

Tags

The Gremlin Agent automatically detects tags from the host system, such as operating system, zone, and local hostname. Gremlin also detects additional tags from certain cloud providers, including Amazon Web Services and Azure. You can also create custom tags and assign them to your Gremlin Agent(s) during installation. See Network Tags for instructions on adding custom tags to Gremlin.

Process Collection

In order to run dependency tests using Gremlin Reliability Management (RM), the Gremlin agent must be configured to collect process data by enabling Process Collection. When Process Collection is enabled, the Gremlin Agent collects process information from hosts, containers, and Kubernetes objects, and uses it to detect dependencies.

To learn more about enabling or disabling process collection, see Process Collection.