LogoLogo
WarpStream.comSlackDiscordContact UsCreate Account
  • Overview
    • Introduction
    • Architecture
      • Service Discovery
      • Write Path
      • Read Path
      • Life of a Request (Simplified)
    • Change Log
  • Getting Started
    • Install the WarpStream Agent / CLI
    • Run the Demo
    • "Hello World" for Apache Kafka
  • BYOC
    • Run the Agents Locally
    • Deploy the Agents
      • Object Storage Configuration
      • Kubernetes Known Issues
      • Rolling Restarts and Upgrades
    • Client Configuration
      • Tuning for Performance
      • Configure Clients to Eliminate AZ Networking Costs
        • Force Interzone Load Balancing
      • Configuring Kafka Client ID Features
      • Known Issues
    • Infrastructure as Code
      • Terraform Provider
      • Helm charts
      • Terraform Modules
    • Monitoring
      • Pre-made Datadog Dashboard
      • Pre-made Grafana Dashboard
      • Important Metrics and Logs
      • Recommended List of Alerts
      • Monitoring Consumer Groups
      • Hosted Prometheus Endpoint
    • Authentication
      • SASL Authentication
      • Mutual TLS (mTLS)
      • Basic Authentication
    • Advanced Agent Deployment Options
      • Agent Roles
      • Agent Groups
      • Protect Data in Motion with TLS Encryption
      • Low Latency Clusters
      • Network Architecture Considerations
      • Agent Configuration Reference
      • Reducing Infrastructure Costs
      • Client Configuration Auto-tuning
    • Hosted Metadata Endpoint
    • Managed Data Pipelines
      • Cookbooks
    • Schema Registry
      • WarpStream BYOC Schema Registry
      • Schema Validation
      • WarpStream Schema Linking
    • Orbit
    • Port Forwarding (K8s)
  • Reference
    • ACLs
    • Billing
      • Direct billing
      • AWS Marketplace
    • Benchmarking
    • Compression
    • Protocol and Feature Support
      • Kafka vs WarpStream Configuration Reference
      • Compacted topics
    • Secrets Overview
    • Security and Privacy Considerations
    • API Reference
      • API Keys
        • Create
        • Delete
        • List
      • Virtual Clusters
        • Create
        • Delete
        • Describe
        • List
        • DescribeConfiguration
        • UpdateConfiguration
      • Virtual Clusters Credentials
        • Create
        • Delete
        • List
      • Monitoring
        • Describe All Consumer Groups
      • Pipelines
        • List Pipelines
        • Create Pipeline
        • Delete Pipeline
        • Describe Pipeline
        • Create Pipeline Configuration
        • Change Pipeline State
      • Invoices
        • Get Pending Invoice
        • Get Past Invoice
    • CLI Reference
      • warpstream agent
      • warpstream demo
      • warpstream cli
      • warpstream cli-beta
        • benchmark-consumer
        • benchmark-producer
        • console-consumer
        • console-producer
        • consumer-group-lag
        • diagnose-record
        • file-reader
        • file-scrubber
      • warpstream playground
    • Integrations
      • Arroyo
      • AWS Lambda Triggers
      • ClickHouse
      • Debezium
      • Decodable
      • DeltaStream
      • docker-compose
      • DuckDB
      • ElastiFlow
      • Estuary
      • Fly.io
      • Imply
      • InfluxDB
      • Kestra
      • Materialize
      • MinIO
      • MirrorMaker
      • MotherDuck
      • Ockam
      • OpenTelemetry Collector
      • ParadeDB
      • Parquet
      • Quix Streams
      • Railway
      • Redpanda Console
      • RisingWave
      • Rockset
      • ShadowTraffic
      • SQLite
      • Streambased
      • Streamlit
      • Timeplus
      • Tinybird
      • Upsolver
    • Partitions Auto-Scaler (beta)
    • Serverless Clusters
    • Enable SAML Single Sign-on (SSO)
    • Trusted Domains
    • Diagnostics
      • GoMaxProcs
      • Small Files
Powered by GitBook
On this page
  • Docker
  • Standalone Binary
  • Diagnosing Connection Issues

Was this helpful?

  1. BYOC

Run the Agents Locally

Instructions on how to run the WarpStream Agent locally for testing / development purposes.

Previous"Hello World" for Apache KafkaNextDeploy the Agents

Last updated 6 months ago

Was this helpful?

First, for your platform.

Once you've installed the WarpStream Agent, you can run the playground command to create an account for local development as well as an ephemeral Kafka cluster and Schema Registry cluster.

Docker

docker run -p 8080 -p 9092:9092 -p 9094:9094 public.ecr.aws/warpstream-labs/warpstream_agent:latest playground

The playground stores data in memory only by default, so any data written will no longer be accessible once the process exits.

Once the docker container is running, there will be a Kafka TCP server listening on port 9092 and a Schema Registry HTTP server listening on port 9094.

This means you're ready to run any application locally that expects to connect to Kafka, and it'll connect to WarpStream instead if you set the bootstrap URL to localhost:9092.

You can also replace the URL of your schema registry clients to localhost:9094 and it'll connect to WarpStream's Schema Registry instead.

If you encounter any problems connecting an application running outside of Docker to the WarpStream agent running inside of Docker, follow .

Standalone Binary

Alternatively, if you installed the standalone WarpStream Agent binary and it is in your PATH, you can just run:

warpstream playground

The playground stores data in memory only by default, so any data written will no longer be accessible once the process exits.

Once that completes, run the following command to test the Kafka connection:

warpstream kcmd -type diagnose-connection -bootstrap-host localhost -bootstrap-port 9092

If that succeeds, then you're ready to run any application locally that expects to connect to Apache Kafka, and it'll connect to WarpStream instead if you set the bootstrap URL to localhost:9092.

If the diagnostic command returns an error, follow the provided instructions to diagnose and fix it.

To test the Schema Registry connection, you can send a request to the server with curl as follows:

curl -X POST "http://localhost:9094/subjects/foo/versions" \
     -H "Content-Type: application/json" \
     -d '{"schema": "{\"type\":\"long\"}"}'

If you receive a valid response, such as {"id":1}, then you're ready to run any application locally that expects to connect to a schema registry by replacing schema registry URL with localhost:9094.

Diagnosing Connection Issues

Once the binary is installed, run the following command to test the connection:

warpstream kcmd -type diagnose-connection -bootstrap-host localhost -bootstrap-port 9092

If that succeeds, then you're ready to run any application locally that expects to connect to Apache Kafka, and it'll connect to WarpStream instead if you set the bootstrap URL to localhost:9092.

If the diagnostic command returns an error, follow the provided instructions to diagnose and fix.

The WarpStream Agent binary ships with a utility for diagnosing connection issues. However, diagnosing connection issues cannot be done in a general purpose manner from within a Docker container. Therefore even if you're running the Agent in a Docker container locally, you'll need to follow our to install the raw WarpStream Agent binary locally before proceeding.

"Installation Script" Agent installation instructions
install the WarpStream Agent
our instructions below for diagnosing connection issues