Distributed Load Testing on AWS Lambda

What you'll learn

  • How to scale out your Artillery tests using built-in AWS Lambda support
  • What AWS resources Artillery creates on your behalf to run your tests
  • Current limitations in AWS Lambda support in Artillery

Overview

This guide describes Artillery's support for running highly-distributed serverless load tests on AWS Lambda.

AWS credentials

To execute tests in AWS Lambda the Artillery CLI makes use of the official AWS SDK (opens in a new tab) to create the resources needed to run your tests (see the AWS Resources seciton for details on what Artillery creates).

The SDK requires AWS credentials (opens in a new tab) to be present to work. Please refer to the official AWS documentation if you don't have one set up already: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html (opens in a new tab)

Please see the IAM Permissions section for details on permissions required to run tests from AWS Lambda.

Running tests from AWS Lambda

To run an existing test script from AWS Lambda, add --platform and --platform-opt flags to the run command.

For example, if you have a test script saved in blitz.yml and you want to run it from eu-west-1 region, run the following command:

artillery run-lambda \
   --region eu-west-1 \
   blitz.yml

How it works

AWS resources created

Artillery will create a number of AWS resources behind the scenes to be able to execute your tests. All resources created by Artillery are serverless and created on-demand. There are no long-running infrastructure components involved.

  • An S3 bucket to store an AWS Lambda deployment package generated from your test script
  • An AWS Lambda function. This function is deleted once the test run completes.
  • An SQS queue for communication between Lambda workers executing your test and the Artillery CLI. This queue is deleted once the test run completes.
  • (Optional) An IAM role named artilleryio-default-lambda-role for Lambda functions running your test. This role uses the official AWS-managed AWSLambdaBasicExecutionRole and AWSLambdaVPCAccessExecutionRole (opens in a new tab) policies, and an additional policy which grants access to the SQS queues created for these tests. If a role with that name already exists, Artillery will use it instead of creating it.

IAM permissions

The AWS profile that the Artillery CLI runs under when you run artillery run needs to have sufficient permissions to be able to create the resources listed above.

An example IAM policy definition is below. The AWS IAM user you're using will need permissions to assume a role which makes use of the policy created with this template.

Note that 123456789000 will need to be replaced with the id of the AWS account you'll be using.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CreateOrGetLambdaRole",
      "Effect": "Allow",
      "Action": ["iam:CreateRole", "iam:GetRole", "iam:PassRole", "iam:AttachRolePolicy"],
      "Resource": "arn:aws:iam::123456789000:role/artilleryio-default-lambda-role-*"
    },
    {
      "Sid": "CreateLambdaPolicy",
      "Effect": "Allow",
      "Action": ["iam:CreatePolicy"],
      "Resource": "arn:aws:iam::123456789000:policy/artilleryio-lambda-policy"
    },
    {
      "Sid": "SQSPermissions",
      "Effect": "Allow",
      "Action": ["sqs:*"],
      "Resource": "arn:aws:sqs:*:123456789000:artilleryio*"
    },
    {
      // ListQueues does cannot be scoped to individual resources
      // https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonsqs.html#amazonsqs-queue
      "Sid": "SQSListQueues",
      "Effect": "Allow",
      "Action": ["sqs:ListQueues"],
      "Resource": "*"
    },
    {
      "Sid": "LambdaPermissions",
      "Effect": "Allow",
      "Action": [
        "lambda:InvokeFunction",
        "lambda:CreateFunction",
        "lambda:DeleteFunction",
        "lambda:GetFunctionConfiguration"
      ],
      "Resource": "arn:aws:lambda:*:123456789000:function:artilleryio-*"
    },
    {
      "Sid": "S3Permissions",
      "Effect": "Allow",
      "Action": [
        "s3:CreateBucket",
        "s3:DeleteObject",
        "s3:GetObject",
        "s3:PutObject",
        "s3:ListBucket",
        "s3:GetLifecycleConfiguration",
        "s3:PutLifecycleConfiguration"
      ],
      "Resource": [
        "arn:aws:s3:::artilleryio-test-data-*",
        "arn:aws:s3:::artilleryio-test-data-*/*"
      ]
    }
  ]
}

Setting up the IAM Policy and Role in AWS

To help you get started, here's a small guide to set up the necessary role and policy in AWS. It's especially important to have this created if you're planning on using Artillery in CI/CD.

IAM Policy: You should first create the policy. You can do that from the AWS UI or CLI. To do it from the UI:

  1. Go to IAM -> Policies and select Create Policy. Select JSON.
  2. Copy the JSON policy from the section above. You’ll need to change the following:
    • Change the AWS account from 123456789000 to your account ID;
    • Remove any comments from the JSON policy, as that is not allowed;
  3. Press Next and give the policy a meaningful name.

IAM Role: After the policy created, you'll need to use that policy in a role:

  1. Go to IAM -> Roles and select Create Role;
  2. Select Custom Trust Policy. You can keep the default generated or customise it based on your organisational needs;
  3. After, on the Add Permissions UI, select the policy created in the previous step;
  4. Finally, press Next and give the role a meaningful name.

That's it! You should now be able to use this role to run Artillery tests in Lambda.

If you're setting up OIDC, for example to interact with Github Actions (opens in a new tab), then you might need to also set a Custom Trust Policy. You can do so by editing the Custom Trust Policy in step 2 with the appropriate policy from the GH Actions guide linked above.

Limitations

AWS Lambda support is in preview. There are some limitations to what's possible, and you may run into bugs. Please report any issues via GitHub issues on https://github.com/artilleryio/artillery/issues (opens in a new tab)

Test duration

Each AWS Lambda is limited to 15 minutes (opens in a new tab) of running time, which means that the entire load test cannot run for longer than 15 minutes at the moment.

A running test cannot be stopped

Once an AWS Lambda starts running, there is no way to stop it. Neither the AWS SDK, nor the AWS Console provide that ability.

This means that once a load test starts, it will run to completion. Be mindful of this, and ramp up load on your applications gradually.

Unavailable Artillery features

The following Artillery features are not supported yet:

  • Only built-in plugin metrics-by-endpoint may be used
  • publish-metrics is currently not compatible with AWS Lambda, due to bundle size limits Added inv2.0.5
  • Custom JS code provided with config.processor cannot yet use any third-party npm packages
  • Can't be used to run Playwright tests
  • --target, and --insecure flags for the run command
  • ensure specs
  • before and after hooks run once in each Lambda worker rather than once per test run

Retention Settings

S3 Lifecycle Configuration

Added inv2.0.5

Objects created in Artillery's s3 bucket in your AWS account have the following lifecycle rules by default:

  • Test Run artifacts (e.g. Lambda function .zip file): 2 days
  • Test Run metadata: 7 days

Questions?

Load Testing at ScaleAWS Fargate