# AWS Lambda Triggers

This page explains how to use WarpStream with AWS's Self Managed Apache Kafka Trigger to ingest data from a WarpStream cluster and process the records as events in Lambda.

First, you'll need a deployed WarpStream cluster. You can follow [our instructions on how to deploy the WarpStream Agents in production](https://docs.warpstream.com/warpstream/agent-setup/deploy), or use our [Fly.io](https://docs.warpstream.com/warpstream/reference/integrations/deploy-warpstream-to-fly.io) or [Railway](https://docs.warpstream.com/warpstream/reference/integrations/railway) templates.

If the WarpStream Agents are deployed outside of your AWS cloud account (using Fly.io or Railway, for example), you'll also want to familiarize yourself with our instruction on [configuring authentication](https://github.com/warpstreamlabs/docs/blob/master/reference/integrations/broken-reference/README.md) for the WarpStream Agents.

Once the cluster is up and running, navigate to the WarpStream console and click the "credentials" button for your virtual cluster.

<figure><img src="https://77315434-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FjB7FxO8ty4EXO4HsQP4E%2Fuploads%2Fgit-blob-7e50246b3e9a7c07b368ee3f3e91c344ff36a024%2FScreenshot%202023-11-29%20at%209.18.53%20PM.png?alt=media" alt=""><figcaption></figcaption></figure>

Next, click the "Create Credentials" button to create a new set of credentials.

<figure><img src="https://77315434-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FjB7FxO8ty4EXO4HsQP4E%2Fuploads%2Fgit-blob-a2fcd5cbcabb0edc7c17a627b9653da1a798a215%2FScreenshot%202023-11-29%20at%209.19.29%20PM.png?alt=media" alt=""><figcaption></figcaption></figure>

Pick a name for your credentials, then submit. The next screen will present you with your username and password. Save those values temporarily, because we're going to store them in AWS secret manager next.

<figure><img src="https://77315434-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FjB7FxO8ty4EXO4HsQP4E%2Fuploads%2Fgit-blob-64ff3de37853bdd0e971fffd17d35e092c3abf05%2FScreenshot%202023-11-29%20at%209.19.43%20PM.png?alt=media" alt=""><figcaption></figcaption></figure>

Go to AWS Secrets Manager in the AWS console and create a new secret.

<figure><img src="https://77315434-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FjB7FxO8ty4EXO4HsQP4E%2Fuploads%2Fgit-blob-d1a8c1ee5b07d21d0e7bc98eed68be2c70c7f25c%2FScreenshot%202023-11-29%20at%209.23.17%20PM.png?alt=media" alt=""><figcaption><p>Make sure you store the key/value pairs as username and password.</p></figcaption></figure>

Once the secret is created, go to AWS Lambda in the AWS console and create a new function.

<figure><img src="https://77315434-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FjB7FxO8ty4EXO4HsQP4E%2Fuploads%2Fgit-blob-ed04ce21fadcc38f2a58dba27589aecdc2cd227e%2FScreenshot%202023-11-29%20at%209.25.47%20PM.png?alt=media" alt=""><figcaption></figcaption></figure>

Edit the code of your lambda function to print out the event.

```javascript
export const handler = async (event) => {
  // TODO implement
  console.log("my event", JSON.stringify(event));
  const response = {
    statusCode: 200,
    body: JSON.stringify(event),
  };
  return response;
};
```

Then click deploy.

<figure><img src="https://77315434-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FjB7FxO8ty4EXO4HsQP4E%2Fuploads%2Fgit-blob-cda62f92c75619a816aef06e6bd0d5c31e119ec1%2FScreenshot%202023-11-29%20at%209.27.44%20PM.png?alt=media" alt=""><figcaption></figcaption></figure>

Next, navigate to the "Configuration" tab and then select "Permissions"

<figure><img src="https://77315434-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FjB7FxO8ty4EXO4HsQP4E%2Fuploads%2Fgit-blob-fb9868a36396c20d2219674f4a11db1ee7d5f9e0%2FScreenshot%202023-11-29%20at%209.32.20%20PM.png?alt=media" alt=""><figcaption></figcaption></figure>

From there, click on the Lambda's role.

<figure><img src="https://77315434-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FjB7FxO8ty4EXO4HsQP4E%2Fuploads%2Fgit-blob-9353c6346bfed8e7b3b5dd489a23f59266710a54%2FScreenshot%202023-11-29%20at%209.32.28%20PM.png?alt=media" alt=""><figcaption></figcaption></figure>

Click "Add permissions" then "Attach policies"

<figure><img src="https://77315434-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FjB7FxO8ty4EXO4HsQP4E%2Fuploads%2Fgit-blob-3ba3534bfdd5ed68b7b47371efd9fe2dfde2ea6e%2FScreenshot%202023-11-29%20at%209.32.35%20PM.png?alt=media" alt=""><figcaption></figcaption></figure>

Search for SecretsManagerReadWrite and click "Add permissions".

<figure><img src="https://77315434-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FjB7FxO8ty4EXO4HsQP4E%2Fuploads%2Fgit-blob-961b9d88cc949abe0cd3daafa0a84223992cec1e%2FScreenshot%202023-11-29%20at%209.32.44%20PM.png?alt=media" alt=""><figcaption></figcaption></figure>

{% hint style="warning" %}
This will give the lambda the ability to read all secrets, so you may want to make the permission more granular specifically to the secret we created above.
{% endhint %}

The permission should be added successfully.

<figure><img src="https://77315434-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FjB7FxO8ty4EXO4HsQP4E%2Fuploads%2Fgit-blob-4a0f9d54e9b841afece67573b53f535f880377f8%2FScreenshot%202023-11-29%20at%209.32.54%20PM.png?alt=media" alt=""><figcaption></figcaption></figure>

Once that's done, navigate back to the Lambda and click "Add trigger".

<figure><img src="https://77315434-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FjB7FxO8ty4EXO4HsQP4E%2Fuploads%2Fgit-blob-cd5a3c799962471860541e018821c77c753603f6%2FScreenshot%202023-11-29%20at%209.28.29%20PM.png?alt=media" alt=""><figcaption></figcaption></figure>

Then select the Kafka source and fill in the bootstrap host, topic name, and authentication configuration. Make sure you use `BASIC_AUTH` as the authentication mechanism, and select the secret we created in the previous steps.

<figure><img src="https://77315434-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FjB7FxO8ty4EXO4HsQP4E%2Fuploads%2Fgit-blob-c448ea1bf3c2a5b707e81c404b657992469dd6f2%2FScreenshot%202023-11-29%20at%209.39.57%20PM.png?alt=media" alt=""><figcaption></figcaption></figure>

<figure><img src="https://77315434-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FjB7FxO8ty4EXO4HsQP4E%2Fuploads%2Fgit-blob-3521e8171bdcbcd5f136fee22b16fbeb0b553a91%2FScreenshot%202023-11-29%20at%209.40.02%20PM.png?alt=media" alt=""><figcaption></figcaption></figure>

Finally, click "Add".

<figure><img src="https://77315434-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FjB7FxO8ty4EXO4HsQP4E%2Fuploads%2Fgit-blob-0e8be3c903f343627a14e216f3c4304da025236d%2FScreenshot%202023-11-29%20at%209.41.08%20PM.png?alt=media" alt=""><figcaption></figcaption></figure>

Now that everything is setup, you can produce data to the topic. In this example, we'll use the [WarpStream Agent binary](https://docs.warpstream.com/warpstream/getting-started/install-the-warpstream-agent) (which includes a CLI tool) to produce data.

{% code overflow="wrap" %}

```bash
warpstream kcmd -bootstrap-host matano-test.fly.dev -tls -username ccun_XXXXXXXXX -password ccp_XXXXXXXXXX -type produce -topic test -records hello,world
```

{% endcode %}

After producing the data, you should be able to see it in the Lambda's cloud watch logs.

<figure><img src="https://77315434-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FjB7FxO8ty4EXO4HsQP4E%2Fuploads%2Fgit-blob-93d6b299bacba2337f23133046e597cb1ac0b877%2FScreenshot%202023-11-29%20at%209.44.21%20PM.png?alt=media" alt=""><figcaption></figcaption></figure>

<figure><img src="https://77315434-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FjB7FxO8ty4EXO4HsQP4E%2Fuploads%2Fgit-blob-93376523efa5575fbc055d2a04bf55e2f365cf8d%2FScreenshot%202023-11-29%20at%209.44.31%20PM.png?alt=media" alt=""><figcaption></figcaption></figure>

<figure><img src="https://77315434-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FjB7FxO8ty4EXO4HsQP4E%2Fuploads%2Fgit-blob-874817fab66e9d1cb6717fc56b8b0a8cff795229%2FScreenshot%202023-11-29%20at%209.44.40%20PM.png?alt=media" alt=""><figcaption></figcaption></figure>

You should also now be able to monitor the state of the AWS Lambda's consumer group in the WarpStream console.

<figure><img src="https://77315434-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FjB7FxO8ty4EXO4HsQP4E%2Fuploads%2Fgit-blob-99a409d3daa3458810a9c790e23c2f28f4d31e71%2FScreenshot%202023-11-29%20at%209.48.37%20PM.png?alt=media" alt=""><figcaption></figcaption></figure>

<figure><img src="https://77315434-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FjB7FxO8ty4EXO4HsQP4E%2Fuploads%2Fgit-blob-9b181f618655c90348e27771234c0f49aa6069a0%2FScreenshot%202023-11-29%20at%209.48.44%20PM.png?alt=media" alt=""><figcaption></figcaption></figure>

<figure><img src="https://77315434-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FjB7FxO8ty4EXO4HsQP4E%2Fuploads%2Fgit-blob-860333863c1284894ce3d29b5812d93e783dba66%2FScreenshot%202023-11-29%20at%209.48.53%20PM.png?alt=media" alt=""><figcaption></figcaption></figure>
