How to use config values stored in AWS

Gremlin

Chaos Engineer

Last Updated: December 17, 2023

Categories: Gremlin

Introduction

This tutorial will provide a walkthrough on setting up and using configuration values located in AWS for configuration in your Gremlin agent installation.

Prerequisites

Before you begin this tutorial, you'll need the following:

• An active AWS account
• A Gremlin account (sign up for a free trial)
• A Gremlin agent installation
• AWS credentials setup on the host running the Gremlin agent

Step 1a - Create config value in AWS SSM

To create a parameter in AWS SSM:

• Navigate to the SSM console (us-west-2 example).
• Navigate to 'Parameter Store' on the left side and click 'Create parameter'.
• Create a name for your parameter, and set the type to be either 'String' or 'SecureString', then enter your config value in the value field. When satisifed with the settings, create the parameter.

Step 1b - Create config value in AWS Secrets Manager

To create a secret in AWS Secrets Manager:

• Navigate to the Secrets Manager console (us-west-2 example).
• Click on "Store a new secret" on the right. When creating a secret, use the "Other type of secret" option, and use the "Plaintext" tab to enter the raw values you wish to use.
  • If you wish, you can use a custom KMS key to encrypt the secret.
• Click 'Next' and set the name and any optional fields, and keep going until the secret is created.

Step 2 - Setup permissions for credentials

Whatever credentials you will have the Gremlin agent use will need the corresponding IAM permissions associated with them.

• For SSM parameters, the ssm:GetParameter action on the parameter resource is required.
• For Secrets Manager secrets, the secretsmanager:GetSecretValue action on the secret resource is required.
• If any of these values are encrypted with a KMS key, the kms:Decrypt action is required on the KMS key resource.

Step 3 - Set Gremlin config values to use ARN

Now in your gremlin configuration, you can set some of the configuration values to use the AWS ARN of the AWS resource you created. When the agent starts, it will reach out to AWS to retrieve the value stored there, and keeps it in memory. Here is an example configuration file with AWS ARNs used to store sensitive values:

1## Gremlin Identifier
2identifier: arn:aws:ssm:us-west-2:123412341234:parameter/gremlin-identifier
3
4## Gremlin Team Id
5team_id: arn:aws:ssm:us-west-2:123412341234:parameter/gremlin-team-id
6
7## Gremlin Team Secret
8#team_secret: arn:aws:secretsmanager:us-west-2:123412341234:secret:gremlin/team-secret-djfgdjkf
9
10## Gremlin Team Certificate
11team_certificate: arn:aws:secretsmanager:us-west-2:123412341234:secret:gremlin/team-cert-ovbndf
12
13## Gremlin Team Certificate
14team_private_key: arn:aws:secretsmanager:us-west-2:123412341234:secret:gremlin/team-key-asdgiovb

Note: When using the Gremlin Helm Chart, you can pass these values like you would with raw secret values (requires chart version 0.12.0 or later):

1helm install gremlin gremlin/gremlin \
2     --namespace gremlin \
3     --set      gremlin.hostPID=true \
4     --set      gremlin.hostNetwork=true \
5     --set      gremlin.collect.processes=true \
6     --set      gremlin.container.driver=any \
7     --set      gremlin.secret.managed=true \
8     --set      gremlin.secret.type=certificate \
9     --set      gremlin.secret.clusterID=${GREMLIN_CLUSTER_ID} \
10     --set      gremlin.secret.teamID=${GREMLIN_TEAM_ID} \
11     --set      gremlin.serviceUrl=${GREMLIN_SERVICE_URL} \
12     --set      gremlin.secret.certificate=arn:aws:secretsmanager:us-west-2:123412341234:secret:gremlin/team-cert-ovbndf \
13     --set      gremlin.secret.key=arn:aws:secretsmanager:us-west-2:123412341234:secret:gremlin/team-key-asdgiovb \
14     --set      gremlin.serviceAccount.annotations."eks\.amazonaws\.com\/role-arn"="arn:aws:iam::123412341234:role/K8sServiceAccountRole" \
15     --set      chao.serviceAccount.annotations."eks\.amazonaws\.com\/role-arn"="arn:aws:iam::123412341234:role/ChaoK8sServiceAccountRole"

Conclusion

You've setup your Gremlin agent to use remote configuration values, increasing the security of your configuration! Refer to the agent configuration to read all the supported values you can use an AWS ARN for.