Configuring the Gremlin Agent
This documentation page shows you how to configure the Gremlin Agent. You can configure Gremlin using either environment variables or a configuration file, although we recommend using the configuration file in most cases. This guide assumes you've already installed and authenticated the Agent.
Using the configuration file
Gremlin's configuration file is located in a file named config.yaml. On Linux, this file lives at /etc/gremlin/config.yaml, and on Windows, its path is C:\ProgramData\Gremlin\Agent\config.yaml. By default, Gremlin ships with a commented-out version of this file. Use the comments or the following example to set up your configuration.
config.yaml require a restart of the gremlind service.Accepted variables
| Variable | Description |
|---|---|
identifier | a string that uniquely identifies a Gremlin installation (defaults to the hosts IP address) |
team_id | the ID of your Gremlin team |
tags | key-value pairs that help you target this machine during experiments |
team_secret | the secret value (used by secret authentication only) |
team_certificate | the public portion of the certificate-pair (required for signature auth only) |
team_private_key | the private portion of the certificate-pair (required for signature auth only) |
https_proxy | the url of a proxy in between Gremlin and api.gremlin.com |
ssl_cert_file | the ssl certificate for your https proxy |
push_metrics | set to false if you would like to disable system metrics used for experiment charting (defaults to true) |
collect_processes | enables/disables Process Collection (as of Linux Agent 2.25.0 default is true) |
iam_role | role the agent uses to retrieve remote cloud config values |
Example: config.yaml
1## Gremlin Identifier; uniquely identifies this machine with Gremlin2## AWS ARNs are also supported (see below section on supported services and credentials)3## (can also set with GREMLIN_IDENTIFIER environment variable)4identifier: gremlin-0156## Gremlin Team Id; you can find this value at https://app.gremlin.com/settings/teams7## AWS ARNs are also supported (see below section on supported services and credentials)8## (can also be set with GREMLIN_TEAM_ID environment variable)9team_id: 11111111-1111-1111-1111-1111111111111011## Gremlin Client Tags; Tag your machine with key-value pairs that help you target this machine during experiments12## (can also set with GREMLIN_CLIENT_TAGS environment variable)13tags:14 service: pet-store15 interface: http1617## Gremlin Team Secret, should not be set when using `team_certificate`+`team_private_key`18## AWS ARNs are also supported (see below section on supported services and credentials)19## (can also set with GREMLIN_TEAM_SECRET environment variable)20#team_secret: 11111111-1111-1111-1111-1111111111112122## Gremlin Team Certificate, should not be set when using `team_secret`.23## Paste certificate content here or a paths to the file (prefixed with `file://`)24## AWS ARNs are also supported (see below section on supported services and credentials)25## (can also set with GREMLIN_TEAM_CERTIFICATE_OR_FILE environment variable)26team_certificate: |27 -----BEGIN CERTIFICATE-----28 111111111111111111111111111111111111111111111111111111111111111129 111111111111111111111111111111111111111111111111111111111111111130 111111111111111111111111111111111111111111111111111111111111111131 111111111111111111111111111111111111111111111111111111111111111132 111111111111111111111111111111111111111111111111111111111111111133 111111111111111111111111111111111111111111111111111111111111111134 111111111111111111111111111111111111111111111111111111111111111135 111111111111111111111111111111111111111111111111111111111111111136 111111111111111111111111111111111111111111111111111111111111111137 11111111111111111111111111111111111111111111111111==38 -----END CERTIFICATE-----3940## Gremlin Team Certificate, should not be set when using `team_secret`.41## Paste certificate content here or a paths to the file (prefixed with `file://`)42## AWS ARNs are also supported (see below section on supported services and credentials)43## (can also set with GREMLIN_TEAM_PRIVATE_KEY_OR_FILE environment variable)44team_private_key: file:///var/lib/gremlin/key.pem4546## HTTPS Proxy, set this when routing outbound Gremlin HTTPS traffic through a proxy47## (can also set with HTTPS_PROXY or https_proxy environment variables)48https_proxy: https://localhost:31284950## SSL CERT FILE, set this when using a https proxy with a self-signed certificate51## Paste certificate content here or a paths to the file (prefixed with `file://`)52## (can also set with SSL_CERT_FILE environment variable)53ssl_cert_file: file:///var/lib/gremlin/proxy_cert.pem5455## Push Metrics, tell Gremlin whether to send system metrics to the control plane for charting the impact of experiments in56## real time. Metrics are only collected during active experiments, and only metrics relevant to the experiment are collected.57## defaults to `true`58## (can also set with PUSH_METRICS environment variable)59push_metrics: true6061## Collect Process Data, data about running processes is sent to Gremlin for service discovery.62## defaults to `true` as of Gremlin Linux Agent 2.25.063collect_processes: true
Restarting the Gremlin Agent
To restart the Gremlin Agent and apply your new changes, run the following commands (depending on your environment):
Linux
1sudo systemctl restart gremlind
Windows
1Restart-Service -Name gremlind
Using environment variables
The daemon supports the following environment variables:
| Environment Variable | Description |
|---|---|
GREMLIN_TEAM_ID | Your Team ID (required for authentication) |
GREMLIN_TEAM_SECRET | Your Team Secret (should only require secret or PEM certificates, not both) |
GREMLIN_TEAM_PRIVATE_KEY_OR_FILE | Your PEM-encoded private key or path to a file containing the private key (required for authentication). Paths should be prefixed with file:// . For example, on Unix: file:///var/lib/gremlin/key.pem , on Windows: file://C:/ProgramData/Gremlin/Agent/key.pem |
GREMLIN_TEAM_CERTIFICATE_OR_FILE | The PEM-encoded public-key certificate or path/filename to the file containing your PEM-encoded public-key certificate (required for authentication). Paths should be prefixed with file://. For example, on Unix: file:///var/lib/gremlin/cert.pem , on Windows: file://C:/ProgramData/Gremlin/Agent/cert.pem |
GREMLIN_IDENTIFIER | Custom name for this client (default as the host's IP address) |
GREMLIN_CLIENT_TAGS | Comma-separated list of custom tags to assign to this client. For example: GREMLIN_CLIENT_TAGS="zone=us-east1,role=mysql,foo=bar" |
The following are optional environment variables that may be required in Docker containerized deployments:
| Environment Variable | Description |
|---|---|
GREMLIN_BYPASS_USERNS_REMAP | Indicates to run the Gremlin Daemon on the same Docker namespace as the host. By default, Gremlin does not launch with a namespace defined. To set this variable, change the value to 1 |
GREMLIN_DOCKER_IMAGE | Indicates the location and version of the Gremlin Docker image to use. This variable can be used to indicate an internal repository. For example: GREMLIN_DOCKER_IMAGE=registry.hub.docker.com/gremlin/gremlin:latest |
GREMLIN_IAM_ROLE | Indicates to run the Gremlin Daemon with the specified role when retrieving remote config values |
net=host | Indicates to run the Gremlin Daemon to use the host network. By default, Gremlin network experiments will not affect the host network for containerized environments. |
pid=host | Indicates to run the Gremlin Daemon to affect host processes. By default, Gremlin process killer will not affect host processes for containerized environments. |
In addition, the following standard Linux environment variables allow proxy configuration:
| Environment Variable | Description |
|---|---|
https_proxy | In the form http[s]://[username:password@]address:port |
SSL_CERT_FILE | Indicates location of the proxy certificates used to authenticate traffic with the proxy. For Example: SSL_CERT_FILE=/etc/ssl/certs/ca-example.crt |
Configuration value priorities
If both the environment variables and the configuration file are used when configuring Gremlin, the environment variable values will take priority. For example if you have the GREMLIN_IDENTIFIER environment variable set to foo and in the config.yaml file the identifier field is set to bar, Gremlin will end up using foo as its identifier when registering with the API.
Remote config values
The Gremlin agent supports using AWS ARNs for the following config values in either ENVVARs or config file:
- identifier
- team_id
- team_secret
- team_certificate
- team_private_key
When remote configuration values are used, Gremlin does not store the corresponding configuration values on the hard disk of the system where it runs. When Gremlin is configured to use Secret Authentication option (i.e. using a team_secret), the agent must store temporary session tokens in the .credentials file, which is only readable by the Gremlin system user.
The following services are supported for reading values:
- ssm
- secretsmanager
The agent requires some credentials to exist on the machine to be able to retrieve the remote values. The credentials preference follows the default AWS order.
Passing environment variables to Systemd or SysV
While configuring the daemon with config.yaml is recommended, process managers like systemd and sysvinit can pass environment variables directly to the daemon process from /etc/default/gremlind. Your installation comes with an example of this file at /etc/default/gremlind.example
1# Gremlin Identifier; uniquely identifies this machine with Gremlin2GREMLIN_IDENTIFIER=gremlin-0134# Gremlin Team ID; you can find this value at https://app.gremlin.com/settings/teams5GREMLIN_TEAM_ID=11111111-1111-1111-1111-11111111111167# `gremlin init` will run automatically when the gremlind service starts.8# Supply extra options to `gremlin init` via this variable9GREMLIN_INIT_OPTS="--tag service=pet-store --tag interface=http"1011# Gremlin Team Secret, should not be set when using `team_certificate`+`team_private_key`12GREMLIN_TEAM_SECRET=11111111-1111-1111-1111-111111111111
You can set any of the environment variables listed in the previous section in the configuration file.
Tags
The Gremlin Agent automatically detects tags from the host system, such as operating system, zone, and local hostname. Gremlin also detects additional tags from certain cloud providers, including Amazon Web Services and Azure. You can also create custom tags and assign them to your Gremlin Agent(s) during installation. See Network Tags for instructions on adding custom tags to Gremlin.
Zone tags are required for the Zone Reliability Test.
Process Collection
In order to run dependency tests using Gremlin Reliability Management (RM), the Gremlin agent must be configured to collect process data by enabling Process Collection. When Process Collection is enabled, the Gremlin Agent collects process information from hosts, containers, and Kubernetes objects, and uses it to detect dependencies.
To learn more about enabling or disabling process collection, see Process Collection.