LogoLogo
WarpStream.comSlackDiscordContact UsCreate Account
  • Overview
    • Introduction
    • Architecture
      • Service Discovery
      • Write Path
      • Read Path
      • Life of a Request (Simplified)
    • Change Log
  • Getting Started
    • Install the WarpStream Agent / CLI
    • Run the Demo
    • "Hello World" for Apache Kafka
  • BYOC
    • Run the Agents Locally
    • Deploy the Agents
      • Object Storage Configuration
      • Kubernetes Known Issues
      • Rolling Restarts and Upgrades
    • Client Configuration
      • Tuning for Performance
      • Configure Clients to Eliminate AZ Networking Costs
        • Force Interzone Load Balancing
      • Configuring Kafka Client ID Features
      • Known Issues
    • Infrastructure as Code
      • Terraform Provider
      • Helm charts
      • Terraform Modules
    • Monitoring
      • Pre-made Datadog Dashboard
      • Pre-made Grafana Dashboard
      • Important Metrics and Logs
      • Recommended List of Alerts
      • Monitoring Consumer Groups
      • Hosted Prometheus Endpoint
    • Authentication
      • SASL Authentication
      • Mutual TLS (mTLS)
      • Basic Authentication
    • Advanced Agent Deployment Options
      • Agent Roles
      • Agent Groups
      • Protect Data in Motion with TLS Encryption
      • Low Latency Clusters
      • Network Architecture Considerations
      • Agent Configuration Reference
      • Reducing Infrastructure Costs
      • Client Configuration Auto-tuning
    • Hosted Metadata Endpoint
    • Managed Data Pipelines
      • Cookbooks
    • Schema Registry
      • WarpStream BYOC Schema Registry
      • Schema Validation
      • WarpStream Schema Linking
    • Orbit
    • Port Forwarding (K8s)
  • Reference
    • ACLs
    • Billing
      • Direct billing
      • AWS Marketplace
    • Benchmarking
    • Compression
    • Protocol and Feature Support
      • Kafka vs WarpStream Configuration Reference
      • Compacted topics
    • Secrets Overview
    • Security and Privacy Considerations
    • API Reference
      • API Keys
        • Create
        • Delete
        • List
      • Virtual Clusters
        • Create
        • Delete
        • Describe
        • List
        • DescribeConfiguration
        • UpdateConfiguration
      • Virtual Clusters Credentials
        • Create
        • Delete
        • List
      • Monitoring
        • Describe All Consumer Groups
      • Pipelines
        • List Pipelines
        • Create Pipeline
        • Delete Pipeline
        • Describe Pipeline
        • Create Pipeline Configuration
        • Change Pipeline State
      • Invoices
        • Get Pending Invoice
        • Get Past Invoice
    • CLI Reference
      • warpstream agent
      • warpstream demo
      • warpstream cli
      • warpstream cli-beta
        • benchmark-consumer
        • benchmark-producer
        • console-consumer
        • console-producer
        • consumer-group-lag
        • diagnose-record
        • file-reader
        • file-scrubber
      • warpstream playground
    • Integrations
      • Arroyo
      • AWS Lambda Triggers
      • ClickHouse
      • Debezium
      • Decodable
      • DeltaStream
      • docker-compose
      • DuckDB
      • ElastiFlow
      • Estuary
      • Fly.io
      • Imply
      • InfluxDB
      • Kestra
      • Materialize
      • MinIO
      • MirrorMaker
      • MotherDuck
      • Ockam
      • OpenTelemetry Collector
      • ParadeDB
      • Parquet
      • Quix Streams
      • Railway
      • Redpanda Console
      • RisingWave
      • Rockset
      • ShadowTraffic
      • SQLite
      • Streambased
      • Streamlit
      • Timeplus
      • Tinybird
      • Upsolver
    • Partitions Auto-Scaler (beta)
    • Serverless Clusters
    • Enable SAML Single Sign-on (SSO)
    • Trusted Domains
    • Diagnostics
      • GoMaxProcs
      • Small Files
Powered by GitBook
On this page
  • Prerequisites
  • Step 1: Create a topic in your WarpStream cluster
  • Step 2: Produce some records
  • Step 3: Set up a Rockset Integration with WarpStream
  • Step 4: Create a Collection from your Integration
  • Step 5: Filter out records with null values for user_id
  • Step 6: Save your Collection
  • Step 7: Produce more messages to your WarpStream topic
  • Next steps

Was this helpful?

  1. Reference
  2. Integrations

Rockset

This page describes how to integrate WarpStream with Rockset, ingest data into Rockset from WarpStream, and query the data in Rockset.

PreviousRisingWaveNextShadowTraffic

Last updated 5 months ago

Was this helpful?

Prerequisites

  1. WarpStream account - get access to WarpStream by registering .

  2. Rockset account - get access to Rockset by registering .

  3. WarpStream cluster up and running.

Step 1: Create a topic in your WarpStream cluster

Obtain the Bootstrap Broker from the WarpStream console by navigating to your cluster and then clicking the Connect tab. If you don't have SASL credentials yet, you can also from the console.

Store these values as environment variables for easy reference:

export BOOTSTRAP_HOST=<YOUR_BOOTSTRAP_BROKER> \
SASL_USERNAME=<YOUR_SASL_USERNAME> \
SASL_PASSWORD=<YOUR_SASL_PASSWORD>;

Then, create a topic using the WarpStream CLI:

warpstream kcmd -bootstrap-host $BOOTSTRAP_HOST -tls -username $SASL_USERNAME -password $SASL_PASSWORD -type create-topic -topic rockset_demo

You should see the following output in your Terminal:

Created topic rockset_demo.

Step 2: Produce some records

Using the WarpStream CLI, produce several messages to your topic:

warpstream kcmd -bootstrap-host $BOOTSTRAP_HOST -tls -username $SASL_USERNAME -password $SASL_PASSWORD -type produce -topic rockset_demo --records '{"action": "click", "user_id": "user_0", "page_id": "home"},,{"action": "hover", "user_id": "user_0", "page_id": "home"},,{"action": "scroll", "user_id": "user_0", "page_id": "home"},,{"action": "click", "user_id": "user_1", "page_id": "home"},,{"action": "click", "user_id": "user_1", "page_id": "home"},,{"action": "click", "user_id": "user_2", "page_id": "home"}'

Note that the WarpStream CLI uses double commas (,,) as a delimiter between JSON records.

Step 3: Set up a Rockset Integration with WarpStream

Click "Start", and fill in the information for your WarpStream cluster.

Click "Save Integration."

Step 4: Create a Collection from your Integration

In the Rockset console, after creating your Integration, click "Create Collection from Integration".

Set the Kafka Topic name to rockset_demo, and set Starting Offset to Earliest. Select JSON as the Data Format. In the Source Preview window, you should see the messages that you produced in Step 2.

Step 5: Filter out records with null values for user_id

In the Rockset Console, write a query in the Ingest Transformation Query Editor that filters out messages with null user_ids.

Step 6: Save your Collection

In the Rockset Console, name your Collection and save it with the default settings.

Step 7: Produce more messages to your WarpStream topic

In your Terminal, produce several more messages:

warpstream kcmd -bootstrap-host $BOOTSTRAP_HOST -tls -username $SASL_USERNAME -password $SASL_PASSWORD -type produce -topic rockset_demo --records '{"action": "click", "user_id": "user_2", "page_id": "home"},,{"action": "hover", "user_id": "user_1", "page_id": "home"},,{"action": "scroll", "user_id": "user_0", "page_id": "home"},,{"action": "click", "user_id": "", "page_id": "home"},,{"action": "click", "user_id": "", "page_id": "home"},,{"action": "click", "user_id": "user_0", "page_id": "home"}'

Look closely! We sent six records to WarpStream, but two of them have null values for user_id.

In the Rockset Console, look at the Summary tab of your Collection overview. You should notice that only four records made it into the Collection from this latest batch. Rockset filtered out the records with null values for user_id!

Next steps

In the Rockset Console, navigate to .

Congrats! Now you know how to integrate WarpStream with Rockset. Next, check out the WarpStream docs on for production, or review the to learn more about what's possible with WarpStream and Rockset.

Integrations > WarpStream
how to configure the Agent
Rockset docs
here
here
create a set of credentials
Be sure to add the port in your Bootstrap Server URL! Your environment variable from Step 1 omits this because the WarpStream CLI defaults to using port 9092 for Kafka requests.