Next-Level Integration Testing with ServerlessSpy

Integration testing for serverless Event-Driven Architecture is as necessary as it is time-consuming. Boilerplate code to write, long waiting for tests completion, tedious debugging. But no more.

Enter ServerlessSpy, a library that eliminates all of the above and makes writing robust integration tests a pleasure. This guide explains how easy it is to use.

Integration Testing is important in serverless – that’s a fact

Much was said about testing serverless microservices. The bottom line is to test in the cloud and follow Testing Diamond / Honeycomb instead of a more traditional Testing Pyramid.

Testing Diamond and Testing Honeycomb put the most pressure on integration testing. This makes total sense for serverless, especially for Event-Driven Architectures with microservices asynchronously communicating through services like EventBridge or SQS. There are plenty of connection points, and automated integration tests are the best way to avoid a headache and tedious debugging if something breaks in the middle.

But how do we implement them?

Integration Testing is also problematic

We have a service. It subscribes to EventBridge Event A, performs Very Important Operations, and some time later sends Event B back to the EventBridge. How do we test it?

No worries, it’s doable. We deploy an extra SQS queue that subscribes to Event B on EventBridge. Then, from our local test runner, we send Event A and poll the SQS, waiting for the result…

We have a service. It’s invoked through API Gateway which triggers a cascade of events resulting in a Very Important Item added to the DynamoDB table. How do we test it?

Again, no worries. We make an HTTP request and then check every few seconds if the item exists in the DynamoDB table. Nice, we didn’t have to create any extra resources; we “only” had to write the code to poll the table periodically!

We have a service… Well, you see where it’s going. Integration testing often requires:

• deploying additional resources to subscribe to events you want to verify,
• writing extra testing boilerplate to poll for the results,
• waiting looong for tests to complete because periodic polling for results always adds latency.

I did all of the above – more than once. But there is a better way.

Integration Testing can be amazing with ServerlessSpy

The ServerlessSpy is a library developed by Marko from Serverless Life for crazy-fast and elegant integration tests.

The ServerlessSpy comes in three parts:

1. A CDK Construct that, in 3 lines of code, creates necessary helper resources, instruments Lambda functions, and generates a list of events we can listen for
2. A type-safe listener for the events with standard Jest matchers for validating event payloads
3. A real-time events monitoring web console, which is a wonderful thing by itself

No boilerplate to write. No extra resources to create on your own.

Listener functions are generated based on the resources available in the CDK stack, so when I say type-safety, I mean no resource names passed as string arguments.

And, last but definitely not least, there is no resource polling. All events are delivered instantly through a WebSocket.

Integration Testing can be as simple as that

For the details on how ServerlessSpy works, it’s best to see the documentation. I’ll just say that using Lambda extension for instrumentation for this use case is really neat.

Our System Under Test

Here, let’s create a sample event-driven service and test it. It will communicate over EventBridge and contain a single Lambda function saving data to a DynamoDB table.

export class ServerlessSpyDemoStack extends cdk.Stack {
	constructor(scope: Construct, id: string, props?: cdk.StackProps) {
		super(scope, id, props);

		const customEventBus = new events.EventBus(this, 'CustomEventBus');
		new cdk.CfnOutput(this, 'CustomEventBusName', {value: customEventBus.eventBusName});

		const dataTable = new dynamodb.Table(this, 'DataTable', {
			billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
			partitionKey: {name: 'pk', type: dynamodb.AttributeType.STRING},
			sortKey: {name: 'sk', type: dynamodb.AttributeType.STRING},
			removalPolicy: cdk.RemovalPolicy.DESTROY,
		});

		const processingLambda = new nodejs.NodejsFunction(this, 'ProcessingLambda', {
			runtime: lambda.Runtime.NODEJS_20_X,
			entry: 'lambda/index.ts',
			timeout: cdk.Duration.seconds(10),
			environment: {
				TABLE_NAME: dataTable.tableName,
				EVENT_BUS_NAME: customEventBus.eventBusName,
			},
		});

		dataTable.grantWriteData(processingLambda);
		processingLambda.addToRolePolicy(new Statement.Events().toPutEvents())

		new events.Rule(this, 'InvokeProcessingLambda', {
			eventBus: customEventBus,
			eventPattern: {
				source: ['ServerlessSpyDemo'],
				detailType: ['EventA'],
			},
			targets: [new eventsTargets.LambdaFunction(processingLambda)],
		});
	}
}

Few resources, nothing fancy. Also, the Processing Lambda source:

import {EventBridgeClient, PutEventsCommand} from "@aws-sdk/client-eventbridge";
import {EventBridgeHandler} from "aws-lambda";
import {DynamoDBClient} from "@aws-sdk/client-dynamodb";
import {DynamoDBDocumentClient, PutCommand} from "@aws-sdk/lib-dynamodb";
import {DynamoDBItem, EventDetail} from "./types";

const ddb = DynamoDBDocumentClient.from(new DynamoDBClient());
const eventBridge = new EventBridgeClient();

export const handler: EventBridgeHandler<string, EventDetail, void> = async (event) => {
	console.log('Received event:', JSON.stringify(event, null, 2));

	await ddb.send(new PutCommand({
		TableName: process.env.TABLE_NAME,
		Item: {
			pk: "shard1",
			sk: event.detail.id,
			message: event.detail.message,
		} satisfies DynamoDBItem,
	}));

	await eventBridge.send(new PutEventsCommand({
		Entries: [
			{
				EventBusName: process.env.EVENT_BUS_NAME,
				Source: "ServerlessSpyDemo",
				DetailType: "EventB",
				Detail: JSON.stringify({id: event.detail.id, message: "Response from Lambda A"} satisfies EventDetail),
			},
		],
	}));
}

export interface EventDetail {
	id: string;
	message: string;
}

export interface DynamoDBItem {
	pk: "shard1",
	sk: string;
	message: string,
}

Now, let’s integrate ServerlessSpy.

Firstly, we add the ServerlessSpy Construct to the stack:

new ServerlessSpy(this, 'ServerlessSpy', {
	generateSpyEventsFileLocation: 'integrationTests/serverlessSpyEvents.ts',
}).spy();

It will create subscribers on our EventBridge and DynamoDB, instrument our Processing Lambda, and generate a list of available events to spy on.

In a real application, we should exclude the ServerlessSpy from deployment on the production environment.

Secondly, we instruct the CDK to save CloudFormation Outputs to a file, as they will contain the WebSocket URL for the ServerlessSpy to connect to and the name of our EventBridge:

{
	...
	"outputsFile": "cdkOutputs.json",
	...
}

We should not commit the cdkOutputs.json file to the repository and add it to .gitignore.

Now, we deploy our stack:

cdk deploy

After a moment, our service will be live and ready. Additionally, we should see a new file generated:

export class ServerlessSpyEvents {
  EventBridgeCustomEventBus: 'EventBridge#CustomEventBus' = 'EventBridge#CustomEventBus';
  DynamoDBDataTable: 'DynamoDB#DataTable' = 'DynamoDB#DataTable';
  FunctionProcessingLambdaRequest: 'Function#ProcessingLambda#Request' = 'Function#ProcessingLambda#Request';
  FunctionProcessingLambdaError: 'Function#ProcessingLambda#Error' = 'Function#ProcessingLambda#Error';
  FunctionProcessingLambdaConsole: 'Function#ProcessingLambda#Console' = 'Function#ProcessingLambda#Console';
  FunctionProcessingLambdaResponse: 'Function#ProcessingLambda#Response' = 'Function#ProcessingLambda#Response';
  EventBridgeRuleCustomEventBusInvokeProcessingLambda: 'EventBridgeRule#CustomEventBus#InvokeProcessingLambda' = 'EventBridgeRule#CustomEventBus#InvokeProcessingLambda';
}

This class lists all events we can spy on. As you can see, we can even listen to Lambda console logs!

The Integration Test itself

What can we test? In a real project, I would just test whether we get a proper event back on the EventBridge. But for the purpose of this demo, let’s go nuts and also verify that the Lambda function is invoked and that it adds an item to the DynamoDB table.

It will look like this:

import {createServerlessSpyListener, ServerlessSpyListener} from "serverless-spy";
import * as cdkOutput from '../cdkOutputs.json';
import {EventBridgeClient, PutEventsCommand} from "@aws-sdk/client-eventbridge";
import {DynamoDBItem, EventDetail} from "../lambda/index";
import {ServerlessSpyEvents} from "./serverlessSpyEvents";

const eventBridge = new EventBridgeClient();

let serverlessSpy: ServerlessSpyListener<ServerlessSpyEvents>;

beforeEach(async () => {
	serverlessSpy = await createServerlessSpyListener<ServerlessSpyEvents>({
		serverlessSpyWsUrl: cdkOutput.ServerlessSpyDemo.ServerlessSpyWsUrl,
	});
});

afterEach(async () => {
	serverlessSpy.stop();
});

test('EventBridge to Lambda to DynamoDB and EventBridge', async () => {
	const id = new Date().toISOString();
	const message = `Integration Test ${id}`;

	console.log(`Test ID: ${id}`);

	await eventBridge.send(new PutEventsCommand({
		Entries: [
			{
				EventBusName: cdkOutput.ServerlessSpyDemo.CustomEventBusName,
				Source: "ServerlessSpyDemo",
				DetailType: "EventA",
				Detail: JSON.stringify({id, message} satisfies EventDetail),
			},
		],
	}));

	(await serverlessSpy.waitForEventBridgeRuleCustomEventBusInvokeProcessingLambda<EventDetail>({
		condition: event => event.detail.id === id,
	}));

	(await serverlessSpy.waitForDynamoDBDataTable<DynamoDBItem>({
		condition: (item) => item.keys.pk === "shard1" && item.keys.sk === id,
	})).toMatchObject({
		eventName: 'INSERT',
		newImage: {
			pk: "shard1",
			sk: id,
			message: message,
		},
	});

	(await serverlessSpy.waitForEventBridgeCustomEventBus<EventDetail>({
		condition: event => event.detailType === "EventB" && event.detail.id === id,
	})).toMatchObject({
		detail: {
			id,
			message: "Response from Lambda A",
		},
	});
});

In beforeEach(), we start the listener. It will receive all events during our test and allow us to verify them.

In lines #27-36, we begin the test by sending EventA to the EventBridge.

Then, we use ServerlessSpy to verify 3 events:

• in lines #38-40, we wait for the Processing Lambda to be invoked from the EventBridge Rule,
• in lines #42-51, we wait for the item to be put in the DynamoDB Data Table, and we verify it looks like expected,
• in lines #53-60, we wait for the EventB to be sent to the EventBridge and verify that it contains the expected message.

Note the condition argument in all three wait functions. It filters out events that do not match the criteria. Thanks to it, multiple tests can run in parallel and not interfere as long as we can distinguish events by some per-test unique property.

Finally, the order of the wait functions does not matter. The ServerlessSpy receives all events from the moment we launch it in beforeEach(), so there are no race conditions or lost events of any kind.

The whole test, with a warmed-up Lambda, takes 1.6 seconds 🚀

Integration Testing with complete visibility

Debugging failing integration tests can be annoying. We need to go through resources or logs, searching for the failing point… Well, no more.

ServerlessSpy comes with a real-time event monitoring web console. We just launch it:

npx sspy --cdkoutput cdkOutputs.json

and observe incoming events:

How awesome is that? And not only for integration tests but also for development in general.

Integration Testing will be easier

By now, you’ve probably guessed I kind of like the ServerlessSpy.

Yes. I think it’s elegant and awesome. And frankly, it deserves much more attention than it got so far. Aren’t people testing their services?! So maybe visit the repository and leave a star?

The ServerlessSpy is being actively developed. It currently supports Lambda, SQS, SNS, EventBridge, DynamoDB, and S3 events. Step Functions and Kinesis are on the roadmap!

You can find the complete source for this demo on GitHub:

✖

Thank you for reading :-)

Join Newsletter and Stay Updated!

Join 460+ subscribers and get:

• 📨 algorithm-independent information about new posts
• 📘 my ebook about Serverless on AWS
• ✅ zero spam

First name

Email

Subscribe

Thank you! Check your email to confirm subscription.

You are already subscribed. Check your spam folder for newsletter emails.

I'm sorry, something went wrong. Please try again later.

One email every few weeks.
Unsubscribe anytime.

✖

Before you leave...

Get Serverless Ebook!

Join the newsletter and get my free ebook about → serverless best practices, → Infrastructure as Code, → AWS services, and → architecture patterns.

First name

Email

Subscribe

Thank you! Check your email to confirm subscription.

You are already subscribed. Check your spam folder for newsletter emails.

I'm sorry, something went wrong. Please try again later.

One email with a new post every few weeks.
Unsubscribe anytime.