Linux

Better error messages for no container driver error messages that can occur during container attacks if the underlying container runtime becomes unreachable. Error messages now include the failures received from each container runtime for which a connection was attempted.

Fixed a bug where Gremlin would sometimes choose the wrong container driver when multiple container runtimes are present, resulting in failed attacks that indicate the targeted container no longer exists.

Removed the file decompression steps that were introduced in 2.39.0 due to the memory overhead this optimization introduced. A future release will optimize container attack provisioning to a more significant degree.

Fixed an incomplete error message when the gremlind process receives API errors from AWS IMDSv2 endpoints.

Updated dependencies

File system resources for Gremlin container attacks are decompressed on startup of the gremlind agent, which reduces gremlind's CPU usage at attack time.

Provided Gremlin has access to a valid AWS credentials chain, it now interprets AWS ARN values in GREMLIN_TEAM_ID, GREMLIN_TEAM_SECRET, GREMLIN_TEAM_CERTIFICATE_OR_FILE, GREMLIN_TEAM_PRIVATE_KEY_OR_FILE. Gremlin supports ARN values from AWS Secrets Manager or AWS Systems Manager Parameter Store. Gremlin can optionally be supplied with GREMLIN_IAM_ROLE to specify a role to assume for the strict purpose of fetching secret values.

More context is added to various error messages

Regression introduced in 2.37.0 where attacks with invalid arguments would end up Lost Communication instead of Failed

Updated dependencies

Fixed a bug where Gremlin would prevent sending arbitrary signals to PID 1. Now, only SIGKILL is prevented, which is unsupported against PID 1 on Linux.

Security patches for: CVE-2023-29403, CVE-2023-44487, CVE-2023-39325, CVE-2023-29406, CVE-2023-39318

Attacks can sometimes fail to notify the Gremlin Control Plane when its connection is impacted by the attack itself. The Gremlin agent now tolerates these failures more often and attempts to resend failed notifications. This fixes attacks that end up in the HaltFaled stage that would otherwise finish in the Successful stage.

Fixed an issue Certificate Expiry attacks against containers would fail when Gremlin was configured with SSL_CERT_FILE

Fixed an issue where important errors from container attacks were not properly forwarded to the Gremlin control plane, leaving execution outputs from failed attacks without helpful troubleshooting information.

Improved the output of the Gremlin Agent validation routine that happens on startup. When validation fails, details about the failure are written to daemon.log

Updated dependencies

Fixed an issue where attacks were incorrectly labeled HaltFailed when Gremlin fails to notify api.gremlin.com during teardown of the network impact.

Fixed a class of issues where Gremlin would not retry requests that failed with transient network errors. This sometimes lead to failing container attacks that should otherwise succeed.

For users running Gremlin in the Docker container runtime, rollbacks against container targets no longer require provisioning a second container instance, which results in faster rollbacks.

Gremlin provides more context to errors stemming from failed http requests to api.gremlin.com.

For users running Gremlin on AWS, more error information is printed to the log file when AWS metadata cannot be retrieved.

Updated dependencies

Fixed an issue where the Gremlin agent would ignore changes to the identifier field in config.yaml if a valid session has already been generated and is not yet expired. On startup, the Gremlin agent will now correctly regenerate a session using the intended identifier value if it detects that its existing session belongs to a different value for identifier.

Fixed an issue with Certificate Expiry experiments against container targets, where the attack process would not have sufficient Linux capabilities (missing DAC_READ_SEARCH). This fix requires helm chart release 0.11.0 (See #86), however all other attacks will continue to work correctly without this chart update.

Updated Certificate Expiry experiments to discover IPv4-mapped IPv6 addresses (e.g., ::FFFF:192.168.1.1) when a CIDR is specified.

Fixed a regression introduced in 2.22.1 where the Process Killer experiment would incorrectly interpret the interval argument as milliseconds, instead of seconds as intended.

Updated dependencies

Running Certificate Expiry experiments against CIDR values (e.g., 10.0.0.0/24) will make several attempts to find an active IP address in use by the target system for evaluating certificate expiration characteristics within the duration specified by the argument --length.

When installed directly on the host and launched with SystemD, Gremlin agent now runs with ambient capabilities (capabilities(7)). File capabilities are no longer set on /usr/bin/gremlin or /usr/sbin/gremlind.

When installed directly on the host, the suid bit is no longer set for installed binaries /usr/bin/gremlin and /usr/sbin/gremlind. Additionally, these binaries are no longer owned by the gremlin linux user, but owned by root instead.

To install Gremlin with file capabilities and gremlin Linux user ownership in accordance with previous Gremlin versions, set the appropriate GREMLIN_INSTALL_ configuration variables at install time: GREMLIN_INSTALL_USER=gremlin GREMLIN_INSTALL_GROUP=gremlin GREMLIN_INSTALL_BIN_MODE=6111 GREMLIN_INSTALL_BIN_CAPABILITIES=1 sudo -E yum install gremlin gremlind. See Customize Gremlin's Linux User and Group

Previously, gremlind would emit snapshots of process and socket data to Gremlin's control plane over 2 minute intervals. This release significantly reduces network overhead for this data as gremlind now batches up process data over 15 minute intervals, deduplicating all network and process data detected over this interval.

Gremlin now uploads discovered process data at a slower rate, reducing network overhead.

Fixed a regression released in 2.31.0 where the gremlin agent would set the Host header to an incorrect value for outgoing requests to the Gremlin control plane. This can lead to authentication failures for some intermediate web proxies that use this host header for authorizing requests.

Errors related to spawning subprocesses now have more detailed information useful for troubleshooting.

IO Errors related to Gremlin container attacks now have more detailed information useful for troubleshooting.

Gremlin provisions fewer file resources for its attack sidecar processes, reducing the time it takes to launch container attacks.

For hostnames supplied to network attacks, Gremlin delegates DNS queries to the operating system. When this query fails, Gremlin now attemps to resolve the name completely within the running process in an attempt to overcome operating system failures. This allows Gremlin network attacks to continue in the face of failed DNS processing.

Fixed a comment in Gremlin's config.yaml which incorrectly stated that collect_processes was disabled by default.

Fixed an out-of-memory error caused by a 3rd party library during process collection.

Fixed a regression instoduced in 2.29.0 where containers for each attack execution incorrectly bind-mounted the file system of every other attack container running on the host. Given enough attack executions running at the same time, a new attack execution container receives a no space left on device error when attempting such mounts, despite space available. Gremlin no longer makes such mounts.

When running the gremlin/gremlin container image, attack containers no longer run in the hostPath mount /var/lib/gremlin. This would produce permission denied errors on systems where this file system is mounted with the noexec flag, such as GKE COS

The Certificate Expiry attack's ipaddress argument now correctly processes CIDR values (e.g. 10.0.0.0/24). When passed, Gremlin will attempt to find an active IP Address in use by the target system and use it for evaluating certificate expiration characteristics.

The gremlin/gremlin Dockerhub image now contains the strace utility as a convenience for operators that cannot install this utility from the internet.

The Blackhole attack skips impact on ingress traffic when it detects third-party ingress traffic manipulation rules, such as those installed by a CNI like cilium. This allows egress impact to be applied without failing the attack with errors like Exclusivity flag on, cannot modify.

Updated dependencies.

Gremlin's calls to getaddrinfo now fallback to TCP when a nameserver replies with a truncated answer. For more info, see musl libc 1.2.4.

Updated dependencies.

Gremlin now tears down the TCP connection pool with api.gremlin.com after successive timeout failures.

Gremlin includes the name of the targeted network interface in execution log events related to applying network impact.

Updated dependencies.

Fixed an issue where Gremlin would not report back to the control plane the detailed error that occurred during a failed attack. Users encountering this bug may see http: 415: 415 in their execution log.

Fixed an issue where gremlin check api would incorrectly report connection failures, including an error message of http 403.

Fixed several instances where errors were suppressed from http interactions made by the Gremlin agent. All failed http interactions now show the method and path of the attempted call, along with descriptive error messages.

Fixed an issue where Gremlin would run requested attack executions in a way that was detached from the original attack request. This leads to the original attack request ending in a LostCommunication stage, while the detached attacks continue to run.

Corrected the ExecStartPre option in the gremlind.service file which resulted in nuisance errors.

Updated dependencies.

Fixed a bug introduced in 2.31.0 where gremlin init would fail unless the environment variable GREMLIN_TRANSPORT=direct was set.

Added support for tag values to be any simple YAML datatype (boolean, integer, float, string). Previously only strings were supported.

Updated dependencies.

Gremlin can now target container and Kubernetes targets, even when those targets lack network access to api.gremlin.com.

All network traffic from Gremlin attack processes are now routed through /var/lib/gremlin/gremlin.sock. To disable this behavior, provide the following environment variable to the Gremlin agent: GREMLIN_TRANSPORT=direct

Fixed an issue that prevented Gremlin from ingesting Azure Tags.

Fixed an issue that made Gremlin validation unreliable.

Updated dependencies.

Addressed an issue where Gremlin agents enabled with GREMLIN_TEAM_SECRET would fail to start when also configured with GREMLIN_TRANSPORT=domain-socket

Gremlin's version command now prints more build information.

Gremlin can now target container and Kubernetes targets, even when those targets lack network access to api.gremlin.com. See more information at Preview: Gremlin in Kubernetes Restricted Networks

Addressed performance issues that were seen with gremlind when collect_processes=true which would lead to high CPU usage and agents becoming IDLE. Symptoms occurred on systems running many processes and active network connections (over 1K of each).

Various metrics around data collection have beed added to the output of gremlin check daemon for benchmarking purposes.

A warning is now supplied in execution logs when the device argument specifies a device that does not exist.

Fixed a regression in 2.30.0 in which network attacks running in a container without targeting a specific network interface failed to have any impact.

Improved the strategy for selecting the target network interfaces.

Multiple network interface attacks are now supported. Details are available in Network device selection.

IP address and network interface data is collected to improve distributed network attacks.

Updated dependencies.

Gremlin Container attacks no longer create a new Linux mount namespace for the attack. Instead, gremlin attack processes now run in the namespace of the gremlind agent. For Kubernetes environments running AppArmor, this release requires a helm chart update.

Updated dependencies.

Fix a bug in collect_certs when the target dropped the network connection before completing the TLS setup.

Updated help URLs.

Updated dependencies.

Add support for containerd builds that do not provide versioning metadata.

Updated dependencies.

Fix a bug that prevented collect_certs from working when run against a container.

Updated dependencies.

Add a short argument (-n) for the not_less_than option.

Updated dependencies.

Fixed an issue affecting Docker CRI on cgroupv2; Gremlin previously failed to roll back network attacks if the target container was killed during the attack.

Gremlin now supports OpenShift 4.9+ and CRI-O 1.22+

Fixed an issue affecting containerd and CRI-O on cgroupv2; Gremlin previously failed to roll back network attacks if the target container was killed during the attack.

Fixed an issue where Gremlin was not resolving internal hostnames in some instances.

Introduce Certificate Expiry test for Reliability Management.

Updated dependencies.

Agent interactions with AWS APIs now use IMDSv2.

Fixed a bug where Gremlin would not properly launch attacks that resolve to a large amount of IP addresses / blocks.

All Gremlin container drivers now work with cgroup2-enabled kernels.

Updated dependencies.

Updated dependencies.

Process Collection is now automatically enabled. Process Collection gathers information about the processes running on Linux machines where the Gremlin Agent is installed to detect system dependencies. To disable Process Collection, see Disable Process Collection.

Updated dependencies.

Fixed a bug where Gremlin's dependency discovery features would not work when IPv6 was disabled.

Fixed a bug where Gremlin would not properly include swap in free memory calculations, leading to incorrect attack results.

Updated dependencies.

Fixed a bug where Gremlin hides informative warnings about its failure to capture dependency discovery data. Now, gremlind logs WARN messages when it fails to find socket data for any given process. Logs are written only once upon first occurrence.

Fixed a bug where Gremlin would attempt to allocate more memory than was available when the cgroup attribute memory.limit_in_bytes was higher than available system memory.

Updated dependencies.

Gremlin's Memory attack now has a new argument: --allocation-strategy (-s), which informs Gremlin on how to interpret other memory consumption arguments: --percent, --mb, and --gb. See more at Memory: Options

Updated dependencies.

Updated dependencies.

Correctly handle proxy usernames and passwords that contain special characters. Special characters must be percent-escaped. For example, %5C is used in place of a backslash. Details are available here.

Updated dependencies.

Disk attack improvements. The Disk attack is much faster, more accurate, and safer.

Updated dependencies.

Updated dependencies

Fixed a bug where Gremlin would fail validation if the DNS lookup of api.gremlin.com failed. This is likely to be a problem in high security environments.

Fixed a bug where Gremlin network attacks would fail with an Illegal match error when supplied with either an IP address argument with an IPv6 address, or a hostname argument that resolved to an IPv6 address, and one of the following:

• a protocol (e.g. TCP), or
• a remote or local port.

Fixed an issue on some operating systems where process collection would fail to parse the target system's kernel version.

Updated dependencies

Removed NTP timestamp from gremlin check os

Updated dependencies

Fixed a bug where Gremlin would crash on some operating systems when process collection was enabled. Gremlin avoids crashing and disables process collection when errors are detected.

Fixed a bug where Gremlin used to skip tag configuration even when it had a valid session. Gremlin now always configures tags on startup as long as it can communicate with the Gremlin control plane.

Some automatic Azure tags were not being correctly read. The azEnvironment, location, name, osType, privateIpAddress, publicIpAddress, sku, vmId, vmSetScaleName, and and zone tags are all automatically read and available for targetting.

Gremlin now reads custom Azure tags associated with the machine and makes those tags available for targeting.

On containerd and cri-o, Gremlin now detects when the process gets killed by the OOMKiller during an attack.

The attack Insight I1000 is now detectable on all container drivers when previously it was only detectable on docker-runc.

Updated Rust version used to build agent in response to CVE-2022-21658

Gremlin now sends I1000 event code for the docker-runc container driver when an attack is interrupted by the Out Of Memory (OOM) manager.

Fixed parsing of Docker container API model when oom_kill_disable is null and log Docker container API model when Gremlin fails to parse it correctly.

Updated dependencies

Introduce IPv6 support (i.e. fe90::60:ff:fe00:1,::1) for network attacks.

On startup, the Gremlin agent now performs some validation on its ability to run a CPU and Latency attack. Validation results are accessible through the Clients API.

The Gremlin agent now outputs more information when Gremlin is killed during container attacks.

Updated dependencies

Fixed a bug present in RPM and DEB packaging where the gremlind service startup script changed ownership of /var/log to gremlin:gremlin on SysvInit enabled systems.

Updated dependencies

Fix a bug where Gremlin would ignore GREMLIN_TEAM_ID and other relevant configuration if a valid session was found at /var/lib/gremlin/.credentials. Now, if the configured teamId value differs from that of the .credentials file, Gremlin re-initiates registration to the control plane.

Gremlin now chooses an appropriate container driver more intelligently. When Gremlin detects more than one container runtime running, it chooses the container driver with the most running containers.

Updated dependencies

The Disk attack has been significantly improved. In most cases it is much faster, more accurate, and safer. It also uses significantly less CPU and RAM when filling disk volumes. The improved version is used when the environment variable GREMLIN_DEUCHAINN_EN1023 is set to true; all other values are treated as false. This environment variable may be ignored or removed in a future version without notice.

Fixed AWS tag ingestion when running Gremlin in a container.

Fixed bug with Gremlin's IO attack cleanup when --mode r or --mode w was used. Previously, Gremlin would try to tear down files that did not exist, leading to attack failures.

Improve messages reported by the Gremlin IO attack, when file-creation errors occur.

Updated dependencies

Changed the way PUSH_METRICS and GREMLIN_COLLECT_PROCESSES boolean configuration variables are evaluated. Previously, any non-empty value other than "0" would evaluate to true (e.g. GREMLIN_COLLECT_PROCESSES=false would evaluate to true). This has been changed to provide expected outcomes: the only values that evaluate true are now "1", "true", and "TRUE", leaving all other values to evaluate to false.

Updated dependencies

Fixed a bug where the Gremlin agent does not properly roll back time travel attacks with an offset of 5 seconds or less.

Fixed a bug where the Gremlin agent does not initialize the containerd-runc container driver when running on a system using the systemd cgroup driver.

The percent argument for Disk attacks now accepts real numbers. For example, --percent 27.5 was previously unsupported.

Gremlin no longer relies on the hostname executable to derive the host's hostname. This is replaced by the gethostname(2) system call.

API interactions made by the Gremlin agent now always send the appropriate Content-Type header value.

Updated dependencies

Updated dependencies

Fixed a bug where child processes beyond immediate children of a container's root process were ignored from the process collection mechanisms that inform service discovery.

Updated dependencies

This update fixes Memory attack bugs. Previously, the amount of memory consumed could deviate significantly from what was requested especially when an attack is run just after I/O operations.

The Memory attack is more "aggressive" in the sense that the memory allocated by Gremlin during the attack is more difficult to swap to disk.

Fixed a bug among all container drivers that use runc, introduced in 2.18.4, where attacks against container and Kubernetes object targets would fail if the targets had memory limit values that exceeded 4 GiB.

Fixed a bug among all container drivers that use runc, where the Memory Gremlin's `Percent` argument would calculate incorrect target memory allocation. Instead of using the container's available/total memory statistics, Gremlin would use the host's, causing Gremlin to try (and fail) to allocate more than is allowed by the target's memory limits.

Fixed a bug that prevented the use of comma-separated values as the input for port arguments to the packet_loss Gremlin.

Updated dependencies.

Gremlin container drivers crio-runc, containerd-runc, and docker-runc requested more Linux capabilities than were actually needed by Gremlin: SETFCAP, AUDIT_WRITE, MKNOD, NET_RAW. Gremlin no longer requests these capabilities when running container attacks.

Gremlin no longer fails to make outgoing HTTP calls to api.gremlin.com if there happens to be directives in /etc/resolv.conf that Gremlin does not understand. Gremlin will now log a message when encountering unknown directives and ignore them.

The Gremlin CLI now has a gremlin check daemon subcommand which reports on the status of any running Gremlin agent, as well whether process collection is enabled.

Gremlin now logs errors encountered when collecting process information. Gremlin logs these errors only the first time they are encountered to reduce log noise.

Updated dependencies.

RPM and DEB package installers for gremlind did not properly honor values for GREMLIN_COLLECT_PROCESSES in /etc/default/gremlind.

Updated dependencies.

Gremlin announces Services Discovery for tracking and improving the reliability of distributed services! This update includes support for Services Discovery for Linux.

RPM and DEB packages have been updated to set the following capabilities on /usr/sbin/gremlind: CAP_SYS_PTRACE and CAP_DAC_READ_SEARCH. The two capabilities are necessary for the daemon to collect process information for Services Discovery. While the capabilities are set at installation time, Gremlin process collection features are disabled by default and can be enabled by changing the agent configuration. Visit Process Collection for more information.

Updated dependencies.

While very rare, getting the current username can fail. When that happens the Gremlin Client would fail to run an attack. Instead, this version resorts to using "unknown" when the username cannot be determined.

Upgrade our Docker image to mitigate a security vulnerability in OpenSSL.

Daemon log file management improvements. Previously, the log file was truncated at midnight. That made troubleshooting difficult. The log file is now rolled when it reaches approximately 1 MiB. Ten compressed log files are kept. With this update the current log file typically captures several days and the compressed log files typically capture a few weeks at a modest cost of approximately 2 MiB of disk space.

Fix a bug in Gremlin's argument parsing for the hostnames and ipaddresses arguments for network attacks.

Improve command-line argument parsing by providing better error messages and catching more edge cases related to illegal inputs.

When enabled, Gremlin process collection now correctly labels child processes of containers as container processes, where they previously were labeled as host processes.

When enabled, Gremlin process collection now records process's active Ipv6 sockets if they can be translated to Ipv4. This is most commonly seen among container processes that are running in their host's network namespace.

Patch a vulnerability in a 3rd party library that posed a variety of memory corruption scenarios, most likely use-after-free.

Improve error messages among network attacks when an invalid network device is supplied. Error message now includes all valid devices.

Drop invalid targeting tags with a warning.

The daemon version is included in the gremlin check report.

Occasionally the Docker version was incorrectly parsed which would result in the classic driver being used for container attacks.

Some agent API traffic is now consolidated into fewer requests.

Some agent API traffic is now gzip-compressed, reducing network overhead on machines where Gremlin is installed.

Patch a vulnerability in a 3rd party library that posed a potential buffer overflow scenario

Patch a vulnerability in a 3rd party library that posed a potential scenario to operate on dangling memory references

You can now specify the SSL_CERT_FILE variable via the config.yml file. See the advanced configuration page for details on how to use it.

Gremlin now properly interprets escaped newline characters \n for values of the GREMLIN_SSL_CERT environment variable.

Gremlin now reports container and process data at a slower rate, down from every 5 seconds during active attacks (and every 10 seconds otherwise) to every 30 seconds. We've found that this data changes much less frequently than is justified for a 5-10 second interval. This should result in significantly reduced network overhead required to run Gremlin.

Updated dependencies

Remove minor bugs from preview-only features

The Gremlin agent now writes a message to daemon.log when attacks finish. This provides observers of this log with an approximation on when attacks have ended.

Updated dependencies

Updated dependencies

Gremlin will now log more information when it receives signals such as TERM. Details include the user and process that sent the signal.

Updated dependencies

Introduced 3 new container drivers: docker-runc, crio-runc, and containerd-runc. With this comes support for new container runtimes: Cri-O and Containerd.

Gremlin's container image now runs solely on Alpine Linux, reducing image size and complexity.

Gremlin now provides full support for the systemd cgroup driver when running any of the new container drivers.

Provide operating system tags for Alpine Linux

Update the expiration date for code signing keys

Updated dependencies

Fixed a bug that omitted previous Gremlin versions from showing up at rpm.gremlin.com

Improved Gremlin's ability to discover Linux distributions that would otherwise yield a tag of os-type: Unknown. Among the previously unknown distributions are Alpine, Amazon, Fedora, and Red Hat Enterprise. These distributions will now properly yield an os-type: Linux tag as well as an os-name tag that appropriately describes the Linux distribution.

Updated dependencies

AWS Availability Zone ID (azid) is available for targeting.

AWS tags are now available for targeting.

Error messages from attack executions resulting in InitializationFailed were missing their error output in the UI. Gremlin now properly reports the error that occurs during initialization

Fix a regression introduced in 2.15.0 which removed Gremlin's Systemd service configuration during re-installs and upgrades. Now, Gremlin properly configures Systemd (or SysvInit) on every installation, re-installation, or upgrade.

Output detailed messages when an attack results in a terminated process.

Filter out clearly invalid data when collecting cloud metadata.

Improve output for gremlin check auth

Fixed a bug introduced in 2.12.25 where Gremlin did not accurately determine when SELinux was enabled for Docker users. This produced incorrect behavior for Gremlin's container attacks, as Gremlin failed to mount /var/lib/gremlin with the Docker volume options :z, resulting in permissions errors.

Gremlin now reports Available Memory for gremlin measure memory

When Gremlin runs in a container, it can now be run under custom SELinux process labels. This allows the privileges that Gremlin requires to run correctly to be granted only to Gremlin and not the rest of a host containers running under the default process label: container_t. Learn more about this on our documentation page, or our Github repo

Miscellaneous work, no functional changes.

Improve error messaging when Gremlin fails to find an IP address for a hostname supplied with the --hostname argument, which can be passed to any network attack. Error message now mentions failures due to specifying a hostname that maps to an invalid DNS record type, such as NS.

Gremlin was not correctly using the SSL_CERT_FILE environment variable when running attacks against containers. As a result, Gremlin would only properly trust intermediate SSL proxies if the file referenced in SSL_CERT_FILE had a path within /var/lib/gremlin. Now, this file can live anywhere on the file system, so long as Gremlin has access to it.

Patch a vulnerability in a 3rd party library that posed a potential denial of service to Gremlin's outbound https connections. In practice this is 100% mitigated unless connecting Gremlin through a malicious SSL proxy

Updated dependencies

Gremlin was not using the custom TLS trust store (specified by the SSL_CERT_FILE environment variable) when carrying out attacks against containers. This resulted in a failure to launch container attacks for users that rely on this configuration.

Improve accuracy of latency measurement when checking Gremlin's connectivity to the control plane using gremlin check api. This measurement now omits the time it takes to initialize the HTTP client used to test connectivity.

Gremlin can be installed with a custom group, user, and/or binary mode. The three optional environment variables GREMLIN_INSTALL_GROUP, GREMLIN_INSTALL_USER, GREMLIN_INSTALL_BIN_MODE are set before running the install to establish the security context. The defaults are unchanged: gremlin, gremlin, 6111.

Added more detail to error messages that occur when Gremlin fails to do a DNS lookup of a hostname. Previously the error message did not include the reason for the lookup failure. An example of the new detail we've added is: failed to lookup address information: Name does not resolve.

Fixed a bug where Time travel attacks were not blocking the NTP port of the target, even when told to do so. Now, specifying --ntp, or checking the Block NTP box in the UI, correctly blocks all traffic to outbound NTP servers. Omitting this option still correctly allows NTP traffic on the target.

Fixed a bug where container attacks (including Kubernetes) were not properly setting attacks to ClientAborted when Gremlin's target is killed. This fix includes displaying more information about Gremlin's status after the target is killed.

Fixed a bug in how the Gremlin Agent reports attack status when Gremlin attacks exit abnormally. In many instances, attacks were incorrectly labeled as LostCommunication when they instead failed to start (Failed), or were killed mid-attack (ClientAborted).

Fixed a bug where the Gremlin Disk attack would not clean up the impact files it created if it was halted from the UI.

Changed the way Debian and RPM installation scripts handle failures when adding Gremlin to the Docker Linux group. Previously, Gremlin would fail and terminate the installation if a docker Linux group was found, but could not add Gremlin to it. Now, a warning is printed instead.

Improved the safety guarantees of the Gremlin Agent when loading attacks from the filesystem. Now, if the Gremlin Agent fails to load any attack state due to IO errors, all attacks will be halted immediately to prevent any unexpected behavior.

We now collect an approximate host boot time, this will aid Gremlin to better recognize unique hosts on your team.

Select a default network interface in more cases (also used when Gremlin identifier isn't specified).

Fix bug that prevented the Gremlin agent from reading attack state for attacks created via the CLI. This was preventing users from halting such attacks from the UI, as well as reading logs from the attack details page.

Remove attack.log files associated with attacks that get rolled back from the CLI through gremlin rollback as well as signals, such as from a Ctrl-C.

Immediately halt and mark the attack as "Initialization Failed" if a Disk Gremlin encounters an IO error while writing the desired amount of bytes.

Updated dependencies

Integrate more thoroughly with the cgroups managed by Kubernetes and Docker. Gremlin container attacks now properly report usage metrics to cAdvisor which is used in Kubernetes monitoring and autoscaling triggers. NOTE: Gremlin currently only supports the cgroupfs cgroup driver. View more information

Cap the --workers (-w) argument for Disk and IO attacks to a maximum value equal to the number of CPUs available to Gremlin. This ensures Gremlin is always busy, and not generating more threads than can be fully utilized by the machine on which Gremlin runs. This also eliminates the possibility that Gremlin will exhaust all threads available to Gremlin, which was observed with very large values supplied to --workers (1024 or higher)

Updated dependencies

Improved handling of invalid auth when running gremlin attack-container

Better organization of output of gremlin check auth, including more information in both the success/error cases

The daemon was not properly halting attacks when it did not have access to it's library directory: /var/lib/gremlin, even though it would allow attacks to run. Attacks are now properly halted.

Users can now supply push_metrics inside config.yaml. This attribute is a boolean value that defaults to true, and is equivalent to the environment variable PUSH_METRICS=1

The daemon was not correctly handling the case when it started up in an un-authed state and relied on gremlin init being run to provide the .credentials file. In particular, it was missing some critical metadata which charting relied upon.

Added subheaders to gremlin check auth to better categorize the output

Read tags supplied in config.yaml

Ship example config.yaml to RPM/DEB packages

Auto-initialize daemon if secret is present and credentials are not present

gremlin measure $TYPE now accepts TYPE in uppercase or lowercase (e.g. gremlin measure cpu). It previously only accepted uppercase.

Kubernetes Pod eviction events triggered by Gremlin resource attacks no longer produce Failed attack states. There is now additional information when Gremlin is killed, and the steps it took to clean up.

Gremlin agents installed into Azure now properly set the publicIpAddress metadata tag (erroneously named publicpAddress in prior versions).

Gremlin now pushes CPU metrics for active attacks. These metrics will be used in charting features that allow you to see Gremlin's effect on your machines in real time. To disable this functionality, add PUSH_METRICS=0 to the configuration for gremlind. No data is collected when attacks aren't running, and only data relevant to the attack is collected.

Gremlin can now compete with the resources dedicated to a container, instead of taking free resources from the host. See more about Gremlin and Cgroups

Running attacks are now halted when the gremlind service is told to shut down from process managers

Memory Gremlin more aggressively touches memory it consumes to better ensure that operating systems don't try to reuse some of it

os-name tag added to clients by default; this value, in combination with os-type, now make up the full os description of the machine (i.e. os-type=Linux + os-name=Ubuntu)

Memory leak collecting measurement data

Ensure capabilities are correctly applied during a rollback

Improvements to I/O and Disk attack targeting and capabilities handling

Better local IP address determination when automatically setting GREMLIN_IDENTIFIER

Improved shutdown handling (SIGINT, SIGTERM, and attack halt)

Updated dependencies

There was a regression in 2.12.25 where host attacks that required capabilities did not properly rollback. This release fixes that.

There was a regression in 2.12.23 where the value of SSL_CERT_FILE was not added to the trust store. That is properly wired into the trust store again

/var/log/gremlin/executions/{guid} was not being cleared on halts - now it is

Shutdown container attack showed an error in the logs, now this case is handled more gracefully

gremlin status was displaying UnknownVariantError in some cases

Updated dependencies

Updated dependencies

Gremlin now interfaces with version 1.24 of Docker's REST API over Unix socket /var/run/docker.sock, instead of indirectly through docker shell commands.

Address startup errors referencing number too large to fit in target type, which happens under certain configurations of the target machine's CPU.

Signal handling and process killer improvements

Better handling for the case when a stateful attack doesn't get a chance to clean up properly within a container

Make file management for Gremlin logs more operating system agnostic

Improve capabilities checking

Updated dependencies

Prevent non-privileged users from acquiring Gremlin secrets if they have command-line access to linux hosts while a container attack is running

Supply the correct DOCKER_API_VERSION to container attacks

Updated dependencies

New `os_type tag added to all new Gremlin clients (e.g. os_type:Linux)

Fixed bug that caused Network Gremlins to fail when attacking two or more processes (including containers) when they shared a network interface.

Improved memory attack performance by as much as four times while limiting the CPU impact.

Recover gracefully from operating system out-of-memory errors.

Minor status message improvements for the memory attack.

Fixed a bug where launching a container attack was not respecting the GREMLIN_BYPASS_USERNS_REMAP environment variable. This should get set only when the Docker namespace remapping feature is being used.

Fixed a bug where Memory Gremlin puts unnecessary strain on getrandom and therefore system entropy.

Updated dependencies

Fixed bug where Gremlin (in Docker only) would log errors about missing directories until it received an attack to run

Fixed a bug where the Gremlin CPU attack would leave too much CPU in the idle and sy states. The CPU attack will now consume the requested amount, using us instead.

Minor improvements to gremlin check functionality

Fixed bug where Gremlin would fail attacks due to a closed HTTP stream

Fixed bug where Gremlin would fail to load attacks under certain circumstances

Improved error messaging around loading authentication configuration

New command gremlin check for diagnostics, check out the docs

Updated dependencies

Improve help text for Blackhole Gremlin arguments about ports

Updated dependencies

Fix bug where Gremlin would create /var/lib/gremlin/.credentials with permissions from the OS umask. Gremlin would then change the mode of the created file before writing to it. Now, Gremlin creates the file with proper permissions, without having to change mode later.

Remove world-readable bit from the /var/log/gremlin directory

Updated dependencies

Fix to Memory Gremlin running in containers - we were allowing the Gremlin to allocate more memory than was given to the target container

Bugfix to Memory Gremlin - we were letting the --percentage option consume more memory than was available

Updated dependencies

Fewer writes by the client to the filesystem, reducing the chance that a Disk Gremlin fails

Explicitly track tearing down successful attacks, so we don't halt attacks too early in the case teardown takes a material amount of time.

Ensure Gremlin sidecars launched in a container have the same GREMLIN_IDENTIFIER as the daemon.

Updated dependencies

Make the Memory attack track its allocation time in the Initializing state.

Updated dependencies

Updated dependencies

Updated dependencies

Updated dependencies

Bugfix for gremlin attack-container CLI command

Fix handling of GREMLIN_CLIENT_TAGS, which were ignored starting in 2.11.6.

Added more trust-store file locations

Updated dependencies

Updated dependencies

Automatically populate client tags when running in Microsoft Azure or Google Cloud

Bugfix for halted attacks which ended in a Lost Communication state (introduced in 2.11.2)

Updated dependencies

Updated dependencies

Automatically populate client tags with instance-id when running on AWS EC2.

Updated dependencies

Resource CPU Attacks can now impact All cores and can consume a percentage of CPU capacity

Network DNS attacks now cache the IP address of the Gremlin Control Plane to avoid the attack from halting prematurely

Proxy details are now hidden in the attack logs on successful calls