Deploying Failure Flags on AWS Lambda

This document will walk you through setting up the Failure Flags agent for Lambda Functions. The Failure Flags agent runs alongside your Function and is responsible for connecting your Function with Gremlin.

Installing the Gremlin-Lambda agent on AWS Lambda

Gremlin-Lambda is a Lambda Extension that you can add to your Lambda Functions. Gremlin-Lambda supports both AMD64/x86_64 and ARM64 architectures. You can learn more about the AWS Lambda Extensions API in the AWS documentation.

Setting required environment variables

You can add the Gremlin-Lambda Extension to any Lambda Function without impacting your function availability or performance. But you do need to add configuration to your environment variables before Gremlin-Lambda will add any value. Configuration comes in via environment variables and or configuration files.

Get started quickly with environment variables only:

• FAILURE_FLAGS_ENABLED must be set to either true or yes or 1 to enable the Failure Flags SDK in your application.
• GREMLIN_LAMBDA_ENABLED must be set to either true or yes or 1 to enable Gremlin-Lambda. If unset or set to any other value Gremlin-Lambda will operate in NOOP mode.
• GREMLIN_TEAM_ID must be set to your Gremlin Team ID. This and other credential material is available through the Gremlin UI.
• GREMLIN_TEAM_CERTIFICATE must be set to your Gremlin Team certificate. Newlines may be preserved using the \n escape characters or omited entirely. This and other credential material is available through the Gremlin UI.
• GREMLIN_TEAM_PRIVATE_KEY must be set to your Gremlin Team private key. Newlines may be preserved using the \n escape characters or omited entirely. This and other credential material is available through the Gremlin UI.

If you want to use configuration files, you'll use environment variables to specify the configuration file:

• FAILURE_FLAGS_ENABLED must be set to either true or yes or 1 to enable the Failure Flags SDK in your application.
• GREMLIN_LAMBDA_ENABLED must be set to either true or yes or 1 to enable Gremlin-Lambda. If unset or set to any other value Gremlin-Lambda will operate in NOOP mode.
• GREMLIN_CONFIG_FILE must be set to the fully-qualified path of a file containing a Gremlin-Lambda configuration file.

Alternatively, you can load configuration files from AWS resources like secretsmanager secrets and ssm parameters:

• FAILURE_FLAGS_ENABLED must be set to either true or yes or 1 to enable the Failure Flags SDK in your application.
• GREMLIN_LAMBDA_ENABLED must be set to either true or yes or 1 to enable Gremlin-Lambda. If unset or set to any other value Gremlin-Lambda will operate in NOOP mode.
• GREMLIN_CONFIG_ARN must be set to an ARN of a resource containing a Gremlin-Lambda configuration file.
• GREMLIN_CONFIG_ROLE must be set to an ARN of a AWS IAM role to use when retrieving the configuration file.

Individual Configuration Values from Files or ARNs

You can configure individual configuration values like GREMLIN_TEAM_CERTIFICATE, GREMLIN_TEAM_PRIVATE_KEY, and GREMLIN_CUSTOM_ROOT_CERTIFICATE to retrieve values from files in your Lambda function or from AWS services using their ARNs. Instead of setting those environment values directly, use their _FILE or _ARN counterparts. Files must be fully qualified paths from the root of the resulting Lambda Function file system (typically something like /var/task/<path from your code root>). This project currently supports secretsmanager secret and ssm paramter ARNs.

When you add the Gremlin-Lambda Extension to your Lambda Function and configure the environment variables correctly, your application will be able to consult that extension for Gremlin experiment configuration. You will be able to find your Function in the Gremlin UI under Agents > Application after you launch your app with the layer configured and you exercise the function.

Once you've added Gremlin-Lambda to your project you can use the Failure Flags library (Node, Python, Java, Go) from your Function code!

Creating a Configuration File with Examples

Gremlin-Lambda uses YAML for file-based configuration. These files contain sensitive credential material and should be stored appropriately. The following is a brief description of each property:

• team_id
• team_certificate
• team_private_key
• labels
• https_proxy
• ssl_cert
• debug
• api_endpoint_url

1## Gremlin Team Id - you can find this value at https://app.gremlin.com/settings/teams
2team_id: ffffffff-ffff-ffff-ffff-ffffffffffff
3
4## Failure Flags Service Labels - Add labels to identify unique deployments.
5labels:
6    datacenter: corp-na1
7    project: columbia
8
9## Debug, set to true for enhanced debug logging to STDOUT
10## Uncomment to enable debugging
11#debug: true
12
13## Gremlin Team Certificate - Paste certificate content here.
14team_certificate: |
15    -----BEGIN CERTIFICATE-----
16    ExampleXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
17    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
18    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
19    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
20    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
21    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
22    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
23    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
24    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
25    XXXXXXXX
26    -----END CERTIFICATE-----
27
28## Gremlin Team Certificate - Paste certificate content here.
29team_private_key: |
30    -----BEGIN EC PRIVATE KEY-----
31    ExampleXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
32    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
33    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==
34    -----END EC PRIVATE KEY-----
35
36## HTTPS Proxy, set this when routing outbound Gremlin HTTPS traffic through a proxy
37#https_proxy: https://corp.proxy.internal:3128
38
39## Custom CA Certificate, set this when using a https proxy with a self-signed certificate
40## Paste certificate content here.
41#ssl_cert: |
42#    -----BEGIN CERTIFICATE-----
43#    ExampleXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
44#    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
45#    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
46#    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
47#    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
48#    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
49#    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
50#    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
51#    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
52#    XXXXXXXX
53#    -----END CERTIFICATE-----
54
55## API ENDPOINT, set this to the Gremlin API endpoint you want to interact with. The beta
56## endpoint is https://beta.gremlin.com/v1. The GA endpoint is at https://api.gremlin.com/v1/ff.
57#api_endpoint_url: https://beta.gremlin.com/v1

Adding the Gremlin-Lambda Extension to your Lambda Function

The Extension's Amazon Resource Name (ARN) varies depending on where your Lambda Function is deployed and which architecture you're using. Retrieve the correct ARN for your architecture and AWS region from this table:

Once you have the correct ARN, add the extension to your Lambda Function by following the instructions in the AWS Lambda Developer Guide.

Building your own Lambda Layer

You may want to build your own Lambda Layer or incorporate the gremlin-lambda binary in a Lambda container image. Maybe you're at risk of using too many layers, or maybe your organization has a package caching policy. Either way you'll want to build your own asset. You can access all of the gremlin-lambda and failure-flags-sidecar builds via our website.

Files provided via assets.gremlin.com have the following format:

https://assets.gremlin.com/packages/gremlin-lambda/<VERSION>/<ARCH>/<FILE>

You can reference the LATEST release at:

https://assets.gremlin.com/packages/gremlin-lambda/latest/<ARCH>/<FILE>

ARCH is one of the following:

• x86_64
• arm64

Different releases may include different files, but the current file set includes:

• gremlin-lambda-linux.tar.gz
• layer-gremlin-lambda.zip
• LICENSE

All versions are listed in a file at: https://assets.gremlin.com/packages/gremlin-lambda/VERSIONS