> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tirdad.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Tirdad Collector

> Stream usage events from Kafka and other sources into Tirdad with reliable delivery

The Tirdad Collector makes it easy to send usage data into Tirdad. It takes care of reading, transforming, and delivering events so you don’t have to worry about scaling or building custom pipelines.

Just plug in your data source, and the collector handles the rest.

## How It Works

The Tirdad Collector is built on [Bento](https://warpstreamlabs.github.io/bento/docs/about), a robust stream processing tool. Bento's strong delivery guarantees and retry capabilities make it ideal for reliable event ingestion.

The collector connects your data sources (Kafka, webhooks, databases) to Tirdad through a processing pipeline with three components:

* **Inputs** read data from various sources (Kafka, webhooks, databases, files)
* **Processors** validate, transform, and filter data to match Tirdad format
* **Outputs** send data to Tirdad API with bulk batching and retry logic

## Key Features

The Tirdad Collector is optimized for high-performance usage metering:

* **Automatic Batching**: Groups events for bulk ingestion (10 events or 2s by default)
* **Smart Routing**: Uses `/events` for single events, `/events/bulk` for batches
* **Retry Logic**: Exponential backoff ensures no data loss on network failures
* **High Throughput**: Handles millions of events per day with minimal latency

## Ingesting Data into Tirdad

The collector always outputs to the Tirdad API:

```yaml theme={null}
output:
  tirdad:
    api_host: https://api.tirdad.ai
    api_key: ${TIRDAD_API_KEY}
    scheme: https
    batching:
      count: 10
      period: 2s
```

Transform your data to match Tirdad event format:

```yaml theme={null}
pipeline:
  processors:
    - mapping: |
        root.event_name = this.event_name
        root.external_customer_id = this.customer_id
        root.properties = this.properties.map_each(p -> p.value.string())
        root.timestamp = this.timestamp.or(now().format_timestamp("2006-01-02T15:04:05Z07:00"))
```

Configure your input source. Example with Kafka:

```yaml theme={null}
input:
  kafka:
    addresses: [${KAFKA_BROKERS}]
    topics: [usage-events]
    consumer_group: tirdad-collector
```

The collector supports 100+ input sources. Learn more in the [Bento documentation](https://warpstreamlabs.github.io/bento/docs/components/inputs/about).

## Installation

The Tirdad Collector is available via:

* **Binaries** from GitHub Releases
* **Container images**
* **Kubernetes** deployment (see Get Started guide)

## Performance & Reliability

The collector provides:

* **Bulk Ingestion**: Batches up to 100 events per API call
* **Automatic Retries**: Failed events retry with exponential backoff
* **At-Least-Once Delivery**: Kafka consumer groups ensure no data loss
* **Observability**: Prometheus metrics and structured logging

Contact [support@tirdad.ai](mailto:support@tirdad.ai) for performance tuning assistance.