This page contains the Apibara documentation as a single document for consumption by LLMs.

---
title: Apibara documentation
titleShort: Overview
description: "Welcome to the Apibara documentation. Find more information about the Apibara protocol."
priority: 1000
fullpage: true
---

# Welcome to the next version of Apibara

This section contains the documentation for the upcoming version of Apibara.
As we move towards a stable release, we will update this documentation to
have more information.


---
title: Installation
description: "Learn how to install and get started with Apibara."
diataxis: tutorial
updatedAt: 2025-03-11
---

# Installation

This tutorial shows how to setup an Apibara project from scratch. The goal is to
start indexing data as quickly as possible and to understand the basic structure
of a project. By the end of this tutorial, you will have a basic indexer that
streams data from two networks (Ethereum and Starknet).

## Installation

This tutorial starts with a fresh Typescript project. In the examples, we use
`pnpm` as the package manager, but you can use any package manager you prefer.

Let's start by creating the project. The `--language` flag specifies which language
to use to implement indexers, while the `--no-create-indexer` flag is used to
delay the creation of the indexer.

:::cli-command

```bash [Terminal]
mkdir my-indexer
cd my-indexer
pnpm dlx apibara@next init . --language="ts" --no-create-indexer
```

```
ℹ Initializing project in .
✔ Created  package.json
✔ Created  tsconfig.json
✔ Created  apibara.config.ts

✔ Project initialized successfully
```

:::

After that, you can install the dependencies.

```bash [Terminal]
pnpm install
```

## Apibara Config

Your indexers' configuration goes in the `apibara.config.ts` file. You can
leave the configuration as is for now.

```typescript [apibara.config.ts]
import { defineConfig } from "apibara/config";

export default defineConfig({
  runtimeConfig: {},
});
```

## EVM Indexer

Let's create the first EVM indexer. All indexers must go in the `indexers`
directory and have a name that ends with `.indexer.ts` or `.indexer.js`.
The Apibara CLI will automatically detect the indexers in this directory and
make them available to the project.

You can use the `apibara add` command to add an indexer to your project. This
command does the following:

- gathers information about the chain you want to index.
- asks about your preferred storage solution.
- creates the indexer.
- adds dependencies to your `package.json`.

:::cli-command

```bash [Terminal]
pnpm apibara add
```

```
✔ Indexer ID: … rocket-pool
✔ Select a chain: › Ethereum
✔ Select a network: › Mainnet
✔ Select a storage: › None
✔ Updated apibara.config.ts
✔ Updated package.json
✔ Created rocket-pool.indexer.ts

ℹ Before running the indexer, run pnpm run install & pnpm run prepare
```

:::

After installing dependencies, you can look at the changes to `apibara.config.ts`.
Notice the indexer's specific runtime configuration. This is a good time to update
the indexing starting block.

```typescript [apibara.config.ts]
import { defineConfig } from "apibara/config";

export default defineConfig({
  runtimeConfig: {
    rocketPool: {
      startingBlock: 21_000_000,
      streamUrl: "https://ethereum.preview.apibara.org",
    },
  },
});
```

Implement the indexer by editing `indexers/rocket-pool.indexer.ts`.

```typescript [rocket-pool.indexer.ts]
import { EvmStream } from "@apibara/evm";
import { defineIndexer } from "@apibara/indexer";

import type { ApibaraRuntimeConfig } from "apibara/types";
import { useLogger } from "@apibara/indexer/plugins";

export default function (runtimeConfig: ApibaraRuntimeConfig) {
  const indexerId = "rocketPool";
  const { startingBlock, streamUrl } = runtimeConfig[indexerId];

  return defineIndexer(EvmStream)({
    streamUrl,
    finality: "accepted",
    startingBlock: BigInt(startingBlock),
    filter: {
      logs: [
        {
          address: "0xae78736Cd615f374D3085123A210448E74Fc6393",
        },
      ],
    },
    plugins: [],
    async transform({ block }) {
      const logger = useLogger();
      const { logs, header } = block;
      logger.log(`Block number ${header?.blockNumber}`);
      for (const log of logs) {
        logger.log(
          `Log ${log.logIndex} from ${log.address} tx=${log.transactionHash}`
        );
      }
    },
  });
}
```

Notice the following:

- The indexer file exports a single indexer.
- The `defineIndexer` function takes the stream as parameter. In this case, the
  `EvmStream` is used. This is needed because Apibara supports multiple networks
  with different data types.
- `streamUrl` specifies where the data comes from. You can connect to streams hosted
  by us, or to self-hosted streams.
- `startingBlock` specifies from which block to start streaming.
- These two properties are read from the `runtimeConfig` object. Use the runtime configuration
  object to have multiple presets for the same indexer.
- The `filter` specifies which data to receive. You can read more about the available
  data for EVM chains in the [EVM documentation](/docs/v2/networks/evm/filter).
- The `transform` function is called for each block. It receives the block as parameter.
  This is where your indexer processes the data.
- The `useLogger` hook returns an indexer-specific logger.

There are more indexer options available, you can find them [in the documentation](/docs/v2/getting-started/indexers).

## Running the indexer

During development, you will use the `apibara` CLI to build and run indexers. For convenience,
the template adds the following scripts to your `package.json`:

```json [package.json]
{
  "scripts": {
    "dev": "apibara dev",
    "build": "apibara build",
    "start": "apibara start"
  }
}
```

- `dev`: runs all indexers in development mode. Indexers are automatically
  reloaded and restarted when they change.
- `build`: builds the indexers for production.
- `start`: runs a _single indexer_ in production mode. Notice you must first
  build the indexers.

Now, run the indexer in development mode.

:::cli-command

```bash [Terminal]
pnpm run dev
```

```
> apibara-app@0.1.0 dev /tmp/my-indexer
> apibara dev

✔ Output directory .apibara/build cleaned
✔ Types written to .apibara/types
✔ Indexers built in 19369 ms
✔ Restarting indexers
rocket-pool | log Block number 21000071
rocket-pool | log Log 239 from 0xae78736cd615f374d3085123a210448e74fc6393 tx=0xe3b7e285c02e9a1dad654ba095ee517cf4c15bf0c2c0adec555045e86ea1de89
rocket-pool | log Block number 21000097
rocket-pool | log Log 265 from 0xae78736cd615f374d3085123a210448e74fc6393 tx=0x8946aaa1ae303a19576d6dca9abe0f774709ff6c3f2de40c11dfda2ab276fbba
rocket-pool | log Log 266 from 0xae78736cd615f374d3085123a210448e74fc6393 tx=0x8946aaa1ae303a19576d6dca9abe0f774709ff6c3f2de40c11dfda2ab276fbba
rocket-pool | log Block number 21000111
rocket-pool | log Log 589 from 0xae78736cd615f374d3085123a210448e74fc6393 tx=0xa01ec6551e76364f6cf687f52823d66b1c07f7a47ce157a9cd9e441691a021f0
...
```

:::

## Starknet indexer

You can index data on different networks in the same project. Let's add an indexer for Starknet.
Like before, you can use the `apibara add` command to add an indexer to your project.

:::cli-command

```bash [Terminal]
pnpm apibara add
```

```
✔ Indexer ID: … strk-staking
✔ Select a chain: › Starknet
✔ Select a network: › Mainnet
✔ Select a storage: › None
✔ Updated apibara.config.ts
✔ Updated package.json
✔ Created strk-staking.indexer.ts

ℹ Before running the indexer, run pnpm run install & pnpm run prepare
```

:::

After that, you can implement the indexer. In this case, the indexer listens for all events
emitted by the STRK staking contract.

Let's start by updating the `apibara.config.ts` file with the starting block.

```typescript [apibara.config.ts]
// ...

export default defineConfig({
  runtimeConfig: {
    // ...
    strkStaking: {
      startingBlock: 900_000,
      streamUrl: "https://starknet.preview.apibara.org",
    },
  },
  // ...
});
```

Then you can implement the indexer.

```typescript [strk-staking.indexer.ts]
import { StarknetStream } from "@apibara/starknet";
import { defineIndexer } from "@apibara/indexer";

import type { ApibaraRuntimeConfig } from "apibara/types";
import { useLogger } from "@apibara/indexer/plugins";

export default function (runtimeConfig: ApibaraRuntimeConfig) {
  const indexerId = "strkStaking";
  const { startingBlock, streamUrl } = runtimeConfig[indexerId];

  return defineIndexer(StarknetStream)({
    streamUrl,
    finality: "accepted",
    startingBlock: BigInt(startingBlock),
    filter: {
      events: [
        {
          address:
            "0x028d709c875c0ceac3dce7065bec5328186dc89fe254527084d1689910954b0a",
        },
      ],
    },
    plugins: [],
    async transform({ block }) {
      const logger = useLogger();
      const { events, header } = block;
      logger.log(`Block number ${header?.blockNumber}`);
      for (const event of events) {
        logger.log(`Event ${event.eventIndex} tx=${event.transactionHash}`);
      }
    },
  });
}
```

You can now run the indexer. In this case, you can specify which indexer you want to run
with the `--indexers` option. When the flag is omitted, all indexers are run concurrently.

:::cli-command

```bash [Terminal]
pnpm run dev --indexers strk-staking
```

```
...
> apibara-app@0.1.0 dev /tmp/my-indexer
> apibara dev "--indexers=strk-staking"

✔ Output directory .apibara/build cleaned
✔ Types written to .apibara/types
✔ Indexers built in 20072 ms
✔ Restarting indexers
strk-staking | log Block number 929092
strk-staking | log Event 233 tx=0x012f8356ef02c36ed1ffddd5252c4f03707166cabcccb49046acf4ab565051c7
strk-staking | log Event 234 tx=0x012f8356ef02c36ed1ffddd5252c4f03707166cabcccb49046acf4ab565051c7
strk-staking | log Event 235 tx=0x012f8356ef02c36ed1ffddd5252c4f03707166cabcccb49046acf4ab565051c7
strk-staking | log Event 236 tx=0x012f8356ef02c36ed1ffddd5252c4f03707166cabcccb49046acf4ab565051c7
strk-staking | log Block number 929119
strk-staking | log Event 122 tx=0x01078c3bb0f339eeaf303bc5c47ea03b781841f7b4628f79bb9886ad4c170be7
strk-staking | log Event 123 tx=0x01078c3bb0f339eeaf303bc5c47ea03b781841f7b4628f79bb9886ad4c170be7
```

:::

## Production build

The `apibara build` command is used to build a production version of the indexer. There are two main
changes for the production build:

- No hot code reloading is available.
- Only one indexer is started. If your project has multiple indexers, it should start them independently.

:::cli-command

```bash [Terminal]
pnpm run build
```

```
> apibara-app@0.1.0 build /tmp/my-indexer
> apibara build

✔ Output directory .apibara/build cleaned
✔ Types written to .apibara/types
◐ Building 2 indexers
✔ Build succeeded!
ℹ You can start the indexers with apibara start
```

:::

Once the indexers are built, you can run them in two (equivalent) ways:

- The `apibara start` command by specifying which indexer to run with the
  `--indexer` flag. In this tutorial we are going to use this method.
- Running `.apibara/build/start.mjs` with Node. This is useful when building Docker images for your indexers.

:::cli-command

```bash [Terminal]
pnpm run start --indexer rocket-pool
```

```
> apibara-app@0.1.0 start /tmp/my-indexer
> apibara start "--indexer" "rocket-pool"

◐ Starting indexer rocket-pool
rocket-pool  | log Block number 21000071
rocket-pool  | log Log 239 from 0xae78736cd615f374d3085123a210448e74fc6393 tx=0xe3b7e285c02e9a1dad654ba095ee517cf4c15bf0c2c0adec555045e86ea1de89
rocket-pool  | log Block number 21000097
rocket-pool  | log Log 265 from 0xae78736cd615f374d3085123a210448e74fc6393 tx=0x8946aaa1ae303a19576d6dca9abe0f774709ff6c3f2de40c11dfda2ab276fbba
rocket-pool  | log Log 266 from 0xae78736cd615f374d3085123a210448e74fc6393 tx=0x8946aaa1ae303a19576d6dca9abe0f774709ff6c3f2de40c11dfda2ab276fbba
rocket-pool  | log Block number 21000111
...
```

:::

## Runtime configuration & presets

Apibara provides a mechanism for indexers to load their configuration from the `apibara.config.ts` file:

- Add the configuration under the `runtimeConfig` key in `apibara.config.ts`.
- Change your indexer's module to return a function that, given the runtime configuration, returns the indexer.

You can update the configuration to define values that are configurable by your indexer. This example used
the runtime configuration to store the DNA stream URL and contract address.

```ts [apibara.config.ts]
import { defineConfig } from "apibara/config";

export default defineConfig({
  runtimeConfig: {
    strkStaking: {
      startingBlock: 900_000,
      streamUrl: "https://starknet.preview.apibara.org",
      contractAddress:
        "0x028d709c875c0ceac3dce7065bec5328186dc89fe254527084d1689910954b0a",
    },
  },
});
```

Then update the indexer to return a function that returns the indexer. Your editor is going to show a type error
since the types of `config.streamUrl` and `config.contractAddress` are unknown, the next session is going to
explain how to solve that issue.

```ts [strk-staking.indexer.ts]
import { StarknetStream } from "@apibara/starknet";
import { defineIndexer } from "@apibara/indexer";
import { useLogger } from "@apibara/indexer/plugins";
import { ApibaraRuntimeConfig } from "apibara/types";

export default function (runtimeConfig: ApibaraRuntimeConfig) {
  const config = runtimeConfig.strkStaking;
  const { startingBlock, streamUrl } = config;

  return defineIndexer(StarknetStream)({
    streamUrl,
    startingBlock: BigInt(startingBlock),
    filter: {
      events: [
        {
          address: config.contractAddress as `0x${string}`,
        },
      ],
    },
    async transform({ block }) {
      // Unchanged.
    },
  });
}
```

### Typescript & type safety

You may have noticed that the CLI generates types in `.apibara/types` before
building the indexers (both in development and production mode).
These types contain the type definition of your runtime configuration. You can
instruct Typescript to use them by adding the following `tsconfig.json` to your
project.

```json [tsconfig.json]
{
  "$schema": "https://json.schemastore.org/tsconfig",
  "compilerOptions": {
    "target": "ES2022",
    "module": "ESNext",
    "moduleResolution": "bundler"
  },
  "include": ["**/*.ts", ".apibara/types"],
  "exclude": ["node_modules"]
}
```

After restarting the Typescript language server you will have a type-safe runtime configuration
right into your indexer!

### Presets

Having a single runtime configuration is useful but not enough for real-world
indexers. The CLI provides a way to have multiple "presets" and select which
one to use at runtime. This is useful, for example, if you're deploying the
same indexers on multiple networks where only the DNA stream URL and contract
addresses change.

You can any number of presets in the configuration and use the `--preset` flag to select which one to use.
For example, you can add a `sepolia` preset that contains the URL of the Starknet Sepolia DNA stream.
If a preset doesn't specify a key, then the value from the root configuration is used.

```ts [apibara.config.ts]
import { defineConfig } from "apibara/config";

export default defineConfig({
  runtimeConfig: {
    streamUrl: "https://starknet.preview.apibara.org",
    contractAddress:
      "0x028d709c875c0ceac3dce7065bec5328186dc89fe254527084d1689910954b0a" as `0x${string}`,
  },
  presets: {
    sepolia: {
      runtimeConfig: {
        streamUrl: "https://starknet-sepolia.preview.apibara.org",
      },
    },
  },
});
```

You can then run the indexer in development mode using the `sepolia` preset.

:::cli-command

```bash [Terminal]
npm run dev -- --indexers=strk-staking --preset=sepolia
```

```
> my-indexer@1.0.0 dev
> apibara dev --indexers=strk-staking --preset=sepolia

✔ Output directory .apibara/build cleaned
✔ Types written to .apibara/types
✔ Indexers built in 3858 ms
✔ Restarting indexers
strk-staking | log Block number 100092
strk-staking | log Event 233 tx=0x012f8356ef02c36ed1ffddd5252c4f03707166cabcccb49046acf4ab565051c7
strk-staking | log Event 234 tx=0x012f8356ef02c36ed1ffddd5252c4f03707166cabcccb49046acf4ab565051c7
strk-staking | log Event 235 tx=0x012f8356ef02c36ed1ffddd5252c4f03707166cabcccb49046acf4ab565051c7
strk-staking | log Event 236 tx=0x012f8356ef02c36ed1ffddd5252c4f03707166cabcccb49046acf4ab565051c7
strk-staking | log Block number 100119
strk-staking | log Event 122 tx=0x01078c3bb0f339eeaf303bc5c47ea03b781841f7b4628f79bb9886ad4c170be7
strk-staking | log Event 123 tx=0x01078c3bb0f339eeaf303bc5c47ea03b781841f7b4628f79bb9886ad4c170be7
...
```

:::

## Storing data & persisting state across restarts

All indexers implemented in this tutorial are stateless. They don't store any
data to a database and if you restart them they will restart indexing from the
beginning.

You can refer to our storage section to learn more about writing data to a database
and persisting the indexer's state across restarts.

- [Drizzle with PostgreSQL](/docs/v2/storage/drizzle-pg)


---
title: Indexers
description: "Learn how to create indexers to stream and transform onchain data."
diataxis: explanation
updatedAt: 2025-01-05
---

# Building indexers

Indexers are created using the `defineIndexer` higher-order function. This function takes a _stream definition_ and returns a function to define the indexer.

The job of an indexer is to stream and process historical data (backfilling) and then switch to real-time mode. Indexers built using our SDK are designed to handle chain-reorganizations automatically.
If, for any reason, you need to receive notifications about reorgs, you can define [a custom `message:invalidate` hook](/docs/v2/getting-started/plugins#hooks) to handle them.

By default, the indexer is stateless (restarts from the beginning on restart) and does not provide any storage. You can add persistence and storage by using one of the provided storage plugins.

### Examples

The following examples show how to create indexers for the Beacon Chain, EVM (Ethereum), and Starknet.

**Beacon Chain indexer**

```ts [beaconchain.indexer.ts]
import { BeaconChainStream } from "@apibara/beaconchain";
import { defineIndexer } from "@apibara/indexer";

export default defineIndexer(BeaconChainStream)({
  /* ... */
});
```

**EVM (Ethereum) indexer**

```ts [evm.indexer.ts]
import { EvmStream } from "@apibara/evm";
import { defineIndexer } from "@apibara/indexer";

export default defineIndexer(EvmStream)({
  /* ... */
});
```

**Starknet indexer**

```ts [starknet.indexer.ts]
import { StarknetStream } from "@apibara/starknet";
import { defineIndexer } from "@apibara/indexer";

export default defineIndexer(StarknetStream)({
  /* ... */
});
```

## With runtime config

To configure the indexer at runtime, export a function that takes the configuration and returns the indexer's definition.

```ts
import { EvmStream } from "@apibara/evm";
import type { ApibaraRuntimeConfig } from "apibara/types";
import { defineIndexer } from "@apibara/indexer";

export default function (runtimeConfig: ApibaraRuntimeConfig) {
  return defineIndexer(EvmStream)({
    // ...
  });
}
```

## Indexer configuration

All indexers take the same configuration options.

- **`streamUrl`**<span class="arg-type">`string`</span><br/><span class="arg-description">The URL of the DNA stream to connect to.</span>
- **`filter`**<span class="arg-type">`TFilter`</span><br/><span class="arg-description">The filter to apply to the DNA stream. This argument is specific to the stream definition. You should refer to the chain's filter reference for the available options (see [Beacon Chain](/docs/v2/networks/beaconchain/filter), [EVM (Ethereum)](/docs/v2/networks/evm/filter), [Starknet](/docs/v2/networks/starknet/filter)).</span>
- **`finality`**<span class="arg-type">`"finalized" | "accepted" | "pending"`</span><br/><span class="arg-description">Receive data with the specified finality. Defaults to `accepted`.</span>
- **`startingCursor`**<span class="arg-type">`{ orderKey: bigint, uniqueKey?: string }`</span><br/><span class="arg-description">The cursor to start the indexer from. Defaults to the genesis block. The `orderKey` represents the block number, and the `uniqueKey` represents the block hash (optional).</span>
- **`debug`**<span class="arg-type">`boolean`</span><br/><span class="arg-description">Enable debug mode. This will print debug information to the console.</span>
- **`transform`**<span class="arg-type">`({ block, cursor, endCursor, finality, context }) => Promise<void>`</span><br/><span class="arg-description">The transform function called for each block received from the DNA stream.</span>
- **`factory`**<span class="arg-type">`({ block, context }) => Promise<{ filter?: TFilter }>`</span><br/><span class="arg-description">The factory function used to add data filters at runtime. Useful for creating indexers for smart contracts like Uniswap V2.</span>
- **`hooks`**<span class="arg-type">`object`</span><br/><span class="arg-description">The hooks to register with the indexer. Refer to the [plugins & hooks](/docs/v2/getting-started/plugins) page for more information.</span>
- **`plugins`**<span class="arg-type">`array`</span><br/><span class="arg-description">The plugins to register with the indexer. Refer to the [plugins & hooks](/docs/v2/getting-started/plugins) page for more information.</span>

### The transform function

The `transform` function is invoked for each block received from the DNA stream. This function is where you should implement your business logic.

**Arguments**

- **`block`**<span class="arg-type">`TBlock`</span><br/><span class="arg-description">The block received from the DNA stream. This is chain-specific (see [Beacon Chain](/docs/v2/networks/beaconchain/data), [EVM (Ethereum)](/docs/v2/networks/evm/data), [Starknet](/docs/v2/networks/starknet/data)).</span>
- **`cursor`**<span class="arg-type">`{ orderKey: bigint, uniqueKey?: string }`</span><br/><span class="arg-description">The cursor of the block before the received block.</span>
- **`endCursor`**<span class="arg-type">`{ orderKey: bigint, uniqueKey?: string }`</span><br/><span class="arg-description">The cursor of the current block.</span>
- **`finality`**<span class="arg-type">`"finalized" | "accepted" | "pending"`</span><br/><span class="arg-description">The finality of the block.</span>
- **`context`**<span class="arg-type">`object`</span><br/><span class="arg-description">The context shared between the indexer and the plugins.</span>

The following example shows a minimal indexer that streams block headers and prints them to the console.

```ts [evm.indexer.ts]
import { EvmStream } from "@apibara/evm";
import { defineIndexer } from "@apibara/indexer";

export default defineIndexer(EvmStream)({
  streamUrl: "https://ethereum.preview.apibara.org",
  filter: {
    header: "always",
  },
  async transform({ block }) {
    const { header } = block;
    console.log(header);
  },
});
```

### The factory function

The `factory` function is used to add data filters at runtime. This is useful for creating indexers for smart contracts that deploy other smart contracts like Uniswap V2 and its forks.

**Arguments**

- **`block`**<span class="arg-type">`TBlock`</span><br/><span class="arg-description">The block received from the DNA stream. This is chain-specific (see [Beacon Chain](/docs/v2/networks/beaconchain/data), [EVM (Ethereum)](/docs/v2/networks/evm/data), [Starknet](/docs/v2/networks/starknet/data)).</span>
- **`context`**<span class="arg-type">`object`</span><br/><span class="arg-description">The context shared between the indexer and the plugins.</span>

The following example shows a minimal indexer that streams `PairCreated` events from Uniswap V2 to detect new pools, and then streams the pool's events.

```ts [uniswap-v2.indexer.ts]
import { EvmStream } from "@apibara/evm";
import { defineIndexer } from "@apibara/indexer";

export default defineIndexer(EvmStream)({
  streamUrl: "https://ethereum.preview.apibara.org",
  filter: {
    logs: [
      {
        /* ... */
      },
    ],
  },
  async factory({ block }) {
    const { logs } = block;
    return {
      /* ... */
    };
  },
  async transform({ block }) {
    const { header, logs } = block;
    console.log(header);
    console.log(logs);
  },
});
```


---
title: Plugins & Hooks
description: "Learn how to use plugins to extend the functionality of your indexers."
diataxis: explanation
updatedAt: 2025-01-05
---

# Plugins & Hooks

Indexers are extensible through hooks and plugins. Hooks are functions that are called at specific points in the indexer's lifecycle. Plugins are components that contain reusable hooks callbacks.

## Hooks

The following hooks are available in all indexers.

 - **`run:before`**<span class="arg-type">`() => void`</span><br/>
   <span class="arg-description">Called before the indexer starts running.</span>
 - **`run:after`**<span class="arg-type">`() => void`</span><br/>
   <span class="arg-description">Called after the indexer has finished running.</span>
 - **`connect:before`**<span class="arg-type">`({ request: StreamDataRequest<TFilter>, options: StreamDataOptions }) => void`</span><br/>
   <span class="arg-description">Called before the indexer connects to the DNA stream. Can be used to change the request or stream options.</span>
 - **`connect:after`**<span class="arg-type">`({ request: StreamDataRequest<TFilter> }) => void`</span><br/>
   <span class="arg-description">Called after the indexer has connected to the DNA stream.</span>
 - **`connect:factory`**<span class="arg-type">`({ request: StreamDataRequest<TFilter>, endCursor: { orderKey: bigint, uniqueKey?: string } }) => void`</span><br/>
   <span class="arg-description">Called before the indexer reconnects to the DNA stream with a new filter (in factory mode).</span>
 - **`message`**<span class="arg-type">`({ message: StreamDataResponse<TBlock> }) => void`</span><br/>
   <span class="arg-description">Called for each message received from the DNA stream. Additionally, message-specific hooks are available: `message:invalidate`, `message:finalize`, `message:heartbeat`, `message:systemMessage`.</span>
 - **`handler:middleware`**<span class="arg-type">`({ use: MiddlewareFunction) => void }) => void`</span><br/>
   <span class="arg-description">Called to register indexer's middlewares.</span>

## Using plugins

You can register plugins in the indexer's configuration, under the `plugins` key.

```ts [my-indexer.indexer.ts]
import { BeaconChainStream } from "@apibara/beaconchain";
import { defineIndexer } from "@apibara/indexer";

import { myAwesomePlugin } from "@/lib/my-plugin.ts";

export default defineIndexer(BeaconChainStream)({
  streamUrl: "https://beaconchain.preview.apibara.org",
  filter: { /* ... */ },
  plugins: [myAwesomePlugin()],
  async transform({ block: { header, validators } }) {
    /* ... */
  },
});
```

## Building plugins

Developers can create new plugins to be shared across multiple indexers or projects. Plugins use the available hooks to extend the functionality of indexers.

The main way to define a plugin is by using the `defineIndexerPlugin` function. This function takes a callback with the indexer as parameter, the plugin should register itself with the indexer's hooks.
When the runner runs the indexer, all the relevant hooks are called.

```ts [my-plugin.ts]
import type { Cursor } from "@apibara/protocol";
import { defineIndexerPlugin } from "@apibara/indexer/plugins";

export function myAwesomePlugin<TFilter, TBlock, TTxnParams>() {
  return defineIndexerPlugin<TFilter, TBlock, TTxnParams>((indexer) => {
    indexer.hooks.hook("connect:before", ({ request, options }) => {
      // Do something before the indexer connects to the DNA stream.
    });

    indexer.hooks.hook("run:after", () => {
      // Do something after the indexer has finished running.
    });
  });
}
```

## Middleware

Apibara indexers support wrapping the `transform` function in middleware. This is used, for example, to wrap all database operations in a transaction.

The middleware is registered using the `handler:middleware` hook. This hook takes a `use` argument to register the middleware with the indexer.
The argument to `use` is a function that takes the indexer's context and a `next` function to call the next middleware or the transform function.

```ts [my-plugin.ts]
import type { Cursor } from "@apibara/protocol";
import { defineIndexerPlugin } from "@apibara/indexer/plugins";

export function myAwesomePlugin<TFilter, TBlock, TTxnParams>() {
  return defineIndexerPlugin<TFilter, TBlock, TTxnParams>((indexer) => {
    const db = openDatabase();
    indexer.hooks.hook("handler:middleware", ({ use }) => {
      use(async (context, next) => {
        // Start a transaction.
        await db.transaction(async (txn) => {
          // Add the transaction to the context.
          context.db = txn;
          try {
            // Call the next middleware or the transform function.
            await next();
          } finally {
            // Remove the transaction from the context.
            context.db = undefined;
          }
        });
      });
    });
  });
}
```

## Inline hooks

For all cases where you want to use a hook without creating a plugin, you can use the `hooks` property of the indexer.

IMPORTANT: inline hooks are the recommended way to add hooks to an indexer. If the same hook is needed in multiple indexers, it is better to create a plugin. Usually, plugins lives in the `lib` folder, for example `lib/my-plugin.ts`.

```ts [my-indexer.indexer.ts]
import { BeaconChainStream } from "@apibara/beaconchain";
import { defineIndexer } from "@apibara/indexer";

export default defineIndexer(BeaconChainStream)({
  streamUrl: "https://beaconchain.preview.apibara.org",
  filter: { /* ... */ },
  async transform({ block: { header, validators } }) {
    /* ... */
  },
  hooks: {
    async "connect:before"({ request, options }) {
      // Do something before the indexer connects to the DNA stream.
    },
  },
});
```

## Indexer lifecycle

The following Javascript pseudocode shows the indexer's lifecycle. This should give you a good understanding of when hooks are called.

```js
function run(indexer) {
  indexer.callHook("run:before");

  const { use, middleware } = registerMiddleware(indexer);

  indexer.callHook("handler:middleware", { use });

  // Create the request based on the indexer's configuration.
  const request = Request.create({
    filter: indexer.filter,
    startingCursor: indexer.startingCursor,
    finality: indexer.finality,
  });

  // Stream options.
  const options = {};

  indexer.callHook("connect:before", { request, options });

  let stream = indexer.streamData(request, options);

  indexer.callHook("connect:after");

  while (true) {
    const { message, done } = stream.next();

    if (done) {
      break;
    }

    indexer.callHook("message", { message });

    switch (message._tag) {
      case "data": {
        const { block, endCursor, finality } = message.data
        middleware(() => {
          if (indexer.isFactoryMode()) {
            // Handle the factory portion of the indexer data.
            // Implementation detail is not important here.
            const newFilter = indexer.factory();
            const request = Request.create(/* ... */);
  
            indexer.callHook("connect:factory", { request, endCursor });
            stream = indexer.streamData(request, options);
          }
  
          indexer.transform({ block, endCursor, finality });
        });
        break;
      }
      case "invalidate": {
        indexer.callHook("message:invalidate", { message });
        break;
      }
      case "finalize": {
        indexer.callHook("message:finalize", { message });
        break;
      }
      case "heartbeat": {
        indexer.callHook("message:heartbeat", { message });
        break;
      }
      case "systemMessage": {
        indexer.callHook("message:systemMessage", { message });
        break;
      }
    }
  }

  indexer.callHook("run:after");
}
```

---
title: Configuration - apibara.config.ts
description: "Learn how to configure your indexers using the apibara.config.ts file."
diataxis: reference
updatedAt: 2025-03-11
---

# apibara.config.ts

The `apibara.config.ts` file is where you configure your project. If the project is using Javascript, the file is named `apibara.config.js`.

## General

**`runtimeConfig: R extends Record<string, unknown>`**

The `runtimeConfig` contains the runtime configuration passed to indexers [if they accept it](/docs/v2/getting-started/indexers#with-runtime-config).
Use this to configure chain or environment specific options such as starting block and contract address.

```ts [apibara.config.ts]
export default defineConfig({
  runtimeConfig: {
    connectionString: process.env["POSTGRES_CONNECTION_STRING"] ?? "memory://",
  },
});
```

**`presets: Record<string, R>`**

Presets represent different configurations of `runtimeConfig`. You can use presets to switch between different environments, like development, test, and production.

```ts [apibara.config.ts]
export default defineConfig({
  runtimeConfig: {
    connectionString: process.env["POSTGRES_CONNECTION_STRING"] ?? "memory://",
  },
  presets: {
    dev: {
      connectionString: "memory://",
    },
  },
});
```

**`preset: string`**

The default preset to use.

**`rootDir: string`**

Change the project's root directory.

**`buildDir: string`**

The directory used for building the indexers. Defaults to `.apibara`.

**`outputDir: string`**

The directory where to output the built indexers. Defaults to `.apibara/build`.

**`indexersDir: string`**

The directory where to look for `*.indexer.ts` or `.indexer.js` files. Defaults to `indexers`.

**`hooks`**

Project level [hooks](/docs/v2/getting-started/plugins).

**`debug: boolean`**

Enable debug mode, printing more detailed logs.

## Build config

**`rolldownConfig: RolldownConfig`**

Override any field in the [Rolldown](https://rolldown.rs/) configuration.

**`exportConditions?: string[]`**

Shorthand for `rolldownConfig.resolve.exportConditions`.

## File watcher

**`watchOptions: WatchOptions`**

Configure Rolldown's file watcher. Defaults to `{ignore: ["**/node_modules/**", "**/.apibara/**"]}`.


---
title: Instrumentation - instrumentation.ts
description: "Learn how to send metrics and traces to your observability platform using instrumentation.ts."
diataxis: reference
updatedAt: 2025-03-19
---

# instrumentation.ts

It's easy to add observability to your indexer using our native instrumentation powered by [OpenTelemetry](https://opentelemetry.io/).

## Convention

To add instrumentation to your project, create a `instrumentation.ts` file at the root (next to `apibara.config.ts`).

This file should export a `register` function, this function is called _once_ by the runtime before the production indexer is started.

```ts [instrumentation.ts]
import type { RegisterFn } from "apibara/types";

export const register: RegisterFn = async () => {
  // Setup OpenTelemetry SDK exporter
};
```

### Logger

You can also replace the default logger by exporting a `logger` function from `instrumentation.ts`. This function should return a `console`-like object with the same methods as `console`.

```ts [instrumentation.ts]
import type { LoggerFactoryFn } from "apibara/types";

export const logger: LoggerFactoryFn = ({ indexer, indexers, preset }) => {
  // Build console here.
};
```

## Examples

### OpenTelemetry with OpenTelemetry Collector

The OpenTelemetry Collector offers a vendor-agnostic implementation for receiving, processing, and exporting telemetry data. Using the collector with Apibara allows you to collect and send both metrics and traces to your preferred observability backend.

#### 1. Install required dependencies

First, install the required OpenTelemetry packages:

```bash [Terminal]
pnpm install @opentelemetry/api @opentelemetry/auto-instrumentations-node @opentelemetry/exporter-metrics-otlp-grpc @opentelemetry/exporter-trace-otlp-grpc @opentelemetry/resources @opentelemetry/sdk-node @opentelemetry/sdk-metrics @opentelemetry/sdk-trace-node @opentelemetry/semantic-conventions
```

#### 2. Update instrumentation.ts to use the OpenTelemetry Collector

Create or update the `instrumentation.ts` file at the root of your project:

```ts [instrumentation.ts]
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-grpc";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-grpc";
import {
  defaultResource,
  resourceFromAttributes,
} from "@opentelemetry/resources";
import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";
import { NodeSDK } from "@opentelemetry/sdk-node";
import { SimpleSpanProcessor } from "@opentelemetry/sdk-trace-node";
import {
  ATTR_SERVICE_NAME,
  ATTR_SERVICE_VERSION,
} from "@opentelemetry/semantic-conventions";
import type { RegisterFn } from "apibara/types";

export const register: RegisterFn = async () => {
  // Create a resource that identifies your service
  const resource = defaultResource().merge(
    resourceFromAttributes({
      [ATTR_SERVICE_NAME]: "apibara",
      [ATTR_SERVICE_VERSION]: "1.0.0",
    })
  );
  const collectorOptions = {
    // configure the grpc endpoint of the opentelemetry-collector
    url: "http://localhost:4317",
  };

  // Configure the OTLP exporter for metrics using grpc protocol,
  const metricExporter = new OTLPMetricExporter(collectorOptions);
  const metricReader = new PeriodicExportingMetricReader({
    exporter: metricExporter,
    exportIntervalMillis: 1000,
  });

  // Configure the OTLP exporter for traces using grpc protocol,
  const traceExporter = new OTLPTraceExporter(collectorOptions);
  const spanProcessors = [new SimpleSpanProcessor(traceExporter)];

  // Configure the SDK
  const sdk = new NodeSDK({
    resource,
    spanProcessors,
    metricReader,
    instrumentations: [getNodeAutoInstrumentations()],
  });

  // Start the SDK
  sdk.start();
};
```

#### 3. Build and run your indexer to see the metrics and traces on the configured OpenTelemetry Collector.

```bash [Terminal]
pnpm build
pnpm start --indexer=your-indexer
```

The following metrics are available out of the box:

- `current_block`: The latest block number being processed by the indexer
- `processed_blocks_total`: Total number of blocks that have been processed
- `reorgs_total`: Number of chain reorganizations detected and handled

Additionally, you can observe detailed traces showing:

- Block processing lifecycle and duration
- Individual transform function execution time
- Chain reorganization handling

### Prometheus

Collecting metrics with Prometheus and visualizing them is a powerful way to monitor your indexers. This section shows how to set up OpenTelemetry with Prometheus.

#### 1. Install required dependencies

First, install the required OpenTelemetry packages:

```bash [Terminal]
pnpm install @opentelemetry/api @opentelemetry/auto-instrumentations-node @opentelemetry/exporter-prometheus @opentelemetry/resources @opentelemetry/sdk-node @opentelemetry/semantic-conventions
```

#### 2. Update instrumentation.ts to use the OpenTelemetry SDK

Update the `instrumentation.ts` file at the root of your project:

```ts [instrumentation.ts]
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { PrometheusExporter } from "@opentelemetry/exporter-prometheus";
import {
  defaultResource,
  resourceFromAttributes,
} from "@opentelemetry/resources";
import { NodeSDK } from "@opentelemetry/sdk-node";
import {
  ATTR_SERVICE_NAME,
  ATTR_SERVICE_VERSION,
} from "@opentelemetry/semantic-conventions";
import type { RegisterFn } from "apibara/types";

export const register: RegisterFn = async () => {
  // Create a resource that identifies your service
  const resource = defaultResource().merge(
    resourceFromAttributes({
      [ATTR_SERVICE_NAME]: "apibara",
      [ATTR_SERVICE_VERSION]: "1.0.0",
    })
  );

  // By default port 9464 will be exposed on the apibara app
  const prometheusExporter = new PrometheusExporter();

  // Configure the SDK
  const sdk = new NodeSDK({
    resource,
    metricReader: prometheusExporter,
    instrumentations: [getNodeAutoInstrumentations()],
  });

  sdk.start();
};
```

#### 3. Build and run your indexer to see the metrics and traces on your configured observability platform.

```bash [Terminal]
pnpm build
pnpm start --indexer=your-indexer
```

### Sentry

#### 1. Install required dependencies

First, install the required Sentry package:

```bash [Terminal]
pnpm install @sentry/node
```

#### 2. Update instrumentation.ts to use Sentry

Update the `instrumentation.ts` file at the root of your project:

```ts [instrumentation.ts]
import * as Sentry from "@sentry/node";
import type { RegisterFn } from "apibara/types";

export const register: RegisterFn = async () => {
  Sentry.init({
    dsn: "__YOUR_DSN__",
    tracesSampleRate: 1.0,
  });
};
```

#### 3. Build and run your indexer with Sentry error tracking enabled

```bash [Terminal]
pnpm build
pnpm start --indexer=your-indexer
```

The Sentry SDK uses OpenTelemetry under the hood, which means any OpenTelemetry instrumentation that emits spans will automatically be picked up by Sentry without additional configuration.

For more information on how to use Sentry with OpenTelemetry, refer to the [Sentry documentation](https://docs.sentry.io/platforms/javascript/guides/node/opentelemetry/).


---
title: Upgrading from v1
description: "Learn how to upgrade your indexers to Apibara v2."
diataxis: how-to
updatedAt: 2025-03-21
---

# Upgrading from v1

This guide explains how to upgrade your Starknet indexers from the old Apibara
CLI experience to the new Apibara v2 experience.

At the time of writing (March 2025), Apibara v2 is still in beta. This means
that it's ready for developers to start testing it, but you should not use it in
production yet.

## Main changes

- The underlying gRPC protocol and data types have changed. You can review all
  the changes [on this page](/docs/v2/networks/starknet/upgrade-from-v1).
- The old CLI has been replaced by a pure Typescript library. This means you
  can now leverage the full Node ecosystem (including Bun and Deno).
- You can now extend indexers with [plugins and hooks](/docs/v2/getting-started/plugins).

## Missing (but planned) features

These features will be added before the final DNA v2 release:

- Add missing integrations like MongoDB.

## Migration

For this guide, we'll assume an indexer like the following:

```ts [indexer.ts]
export const config = {
  streamUrl: "https://mainnet.starknet.a5a.ch",
  startingBlock: 800_000,
  network: "starknet",
  finality: "DATA_STATUS_ACCEPTED",
  filter: {
    header: {},
  },
  sinkType: "console",
  sinkOptions: {},
};

export default function transform(block) {
  return block;
}
```

### Step 1: initialize the Node project

Initialize the project to contain a `package.json` file:

```bash [Terminal]
npm init -y
```

Create the `indexers/` folder where all the indexers will live:

```bash [Terminal]
mkdir indexers
```

Add the dependencies needed to run the indexer. If you're using any external
dependencies, make sure to add them.

:::cli-command

```bash [Terminal]
npm add --save apibara@next @apibara/protocol@next @apibara/indexer@next @apibara/starknet@next
```

```
added 325 packages, and audited 327 packages in 11s

73 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
```

:::

### Step 2: initialize the Apibara project

Create a new file called `apibara.config.ts` in the root of your project.

```ts [apibara.config.ts]
import { defineConfig } from "apibara/config";

export default defineConfig({});
```

### Step 3: update the indexer

Now it's time to update the indexer.

- Move the indexer to the `indexers/` folder, ensuring tha the file name ends
  with `.indexer.ts`.
- Wrap the indexer in a `defineIndexer(StarknetStream)({ /* ... */ })` call.
  Notice that now the stream configuration and transform function live in the same
  configuration object.
- `startingBlock` is now a `BigInt`.
- `streamUrl` is the same, but the URL is different while in beta.
- `finality` is now simpler to type.
- The `filter` object changed. Please refer to the [filter documentation](/docs/v2/networks/starknet/filter) for more information.
- `sinkType` and `sinkOptions` are gone.
- The `transform` function now takes named arguments, with `block` containing the block data.

The following `git diff` shows the changes to the indexer at the beginning of the guide.

```diff
diff --git a/simple.ts b/indexers/simple.indexer.ts
index bb09fdc..701a494 100644
--- a/simple.ts
+++ b/indexers/simple.indexer.ts
@@ -1,15 +1,18 @@
-export const config = {
-    streamUrl: "https://mainnet.starknet.a5a.ch",
-    startingBlock: 800_000,
-    network: "starknet",
-    finality: "DATA_STATUS_ACCEPTED",
+import { StarknetStream } from "@apibara/starknet";
+import { defineIndexer } from "@apibara/indexer";
+import { useLogger } from "@apibara/indexer/plugins";
+
+export default defineIndexer(StarknetStream)({
+    streamUrl: "https://starknet.preview.apibara.org",
+    startingBlock: 800_000n,
+    finality: "accepted",
     filter: {
-        header: {},
+        header: "always",
     },
-    sinkType: "console",
-    sinkOptions: {},
-};
-
-export default function transform(block) {
-    return block;
-}
\ No newline at end of file
+    async transform({ block }) {
+        const logger = useLogger();
+        logger.info(block);
+    },
+});
\ No newline at end of file
```

### Step 4: writing data

In version 1, the indexer would write data returned by `transform` to a sink.
Now, you use plugins to write data to databases like PostgreSQL or MongoDB.

Refer to the plugins documentation for more information.


---
title: Drizzle with PostgreSQL
description: "Store your indexer's data to PostgreSQL using Drizzle ORM."
diataxis: reference
updatedAt: 2025-03-10
---

# Drizzle with PostgreSQL

The Apibara Indexer SDK supports Drizzle ORM for storing data to PostgreSQL.

## Installation

### Using the CLI

You can add an indexer that uses Drizzle for storage by selecting "PostgreSQL"
in the "Storage" section when creating an indexer.

The CLI automatically updates your `package.json` to add all necessary dependencies.

### Manually

To use Drizzle with PostgreSQL, you need to install the following dependencies:

```bash [Terminal]
npm install drizzle-orm pg @apibara/plugin-drizzle@next
```

We recommend using Drizzle Kit to manage the database schema.

```bash [Terminal]
npm install --save-dev drizzle-kit
```

## Schema configuration

You can use the `pgTable` function from `drizzle-orm/pg-core` to define the schema,
no changes required.

The only important thing to notice is that your table must have an `id` column (name configurable)
that uniquely identifies each row. This requirement is necessary to handle chain reorganizations.
Read more how the plugin handles chain reorganizations[on the internals page](/docs/v2/storage/drizzle-pg/internals).

```ts [lib/schema.ts]
import { bigint, pgTable, text, uuid } from "drizzle-orm/pg-core";

export const transfers = pgTable("transfers", {
  id: uuid("id").primaryKey().defaultRandom(),
  amount: bigint("amount", { mode: "number" }),
  transactionHash: text("transaction_hash"),
});
```

## Adding the plugin to your indexer

Add the `drizzleStorage` plugin to your indexer's `plugins`. Notice the following:

- Use the `drizzle` helper exported by `@apibara/plugin-drizzle` to create a
  drizzle instance. This method supports creating an in-memory database (powered by PgLite)
  by specifying the `memory:` connection string.
- Always specify the database schema. This schema is used by the indexer to know which tables
  it needs to protect against chain reorganizations.

```ts [my-indexer.indexer.ts]
import {
  drizzle,
  drizzleStorage,
  useDrizzleStorage,
} from "@apibara/plugin-drizzle";

import { transfers } from "@/lib/schema";

const db = drizzle({
  schema: {
    transfers,
  },
  connectionString: process.env.DATABASE_URL,
});

export default defineIndexer(EvmStream)({
  // ...
  plugins: [drizzleStorage({ db })],
  // ...
});
```

## Writing and reading data from within the indexer

Use the `useDrizzleStorage` hook to access the current database transaction.
This transaction behaves exactly like a regular Drizzle ORM transaction because
it is. Thanks to the way the plugin works and handles chain reorganizations, it
can expose the full Drizzle ORM API without any limitations.

```ts [my-indexer.indexer.ts]
export default defineIndexer(EvmStream)({
  // ...
  async transform({ endCursor, block, context, finality }) {
    const { db } = useDrizzleStorage();

    for (const event of block.events) {
      await db.insert(transfers).values(decodeEvent(event));
    }
  },
});
```

You are not limited to inserting data, you can also update and delete rows.

### Drizzle query

Using the [Drizzle Query interface](https://orm.drizzle.team/docs/rqb) is easy.
Pass the database instance to `useDrizzleStorage`: in this case the database type
is used to automatically deduce the database schema.

**Note**: the database instance is _not_ used to query data but only for type inference.

```ts [my-indexer.indexer.ts]
const database = drizzle({ schema, connectionString });

export default defineIndexer(EvmStream)({
  // ...
  async transform({ endCursor, block, context, finality }) {
    const { db } = useDrizzleStorage(database);

    const existingToken = await db.query.tokens.findFirst({ address });
  },
});
```

## Querying data from outside the indexer

You can query data from your application like you always do, using the standard Drizzle ORM library.

## Improving performance

You should always add an index on the `id` column. This column is used every time the indexer handles
a chain reorganization or a new pending block.

## Database migrations

There are two strategies you can adopt for database migrations:

- run migrations separately, for example using the drizzle-kit CLI.
- run migrations automatically upon starting the indexer.

If you decide to adopt the latter strategy, use the `migrations` option:

```ts [my-indexer.indexer.ts]
import { drizzle } from "@apibara/plugin-drizzle";

const database = drizzle({ schema, connectionString });

export default defineIndexer(EvmStream)({
  // ...
  plugins: [
    drizzleStorage({
      db,
      migrate: {
        // Path relative to the project's root.
        migrationsFolder: "./migrations",
      },
    }),
  ],
  // ...
});
```


---
title: Drizzle's plugin internals
description: "Store your indexer's data to PostgreSQL using Drizzle ORM."
diataxis: reference
updatedAt: 2025-03-10
---

# Drizzle's plugin internals

This section describes how the Drizzle plugin works. Understanding the content of this page is not needed for using the plugin.

## Drizzle and the indexer

The plugin wraps all database operations in the `transform` and `factory` functions in a database transaction.
This ensures that the indexer's state is always consistent and that data is never lost due to crashes or network failures.

More specifically, the plugin is implemented as a [middleware](/docs/v2/getting-started/plugins#middleware).

At a very high level, the plugin looks like the following:

```ts
indexer.hooks.hook("handler:middleware", async ({ use }) => {
  use(async (context, next) => {
    await db.transaction(async (txn) => {
      // Assign the transaction to the context, to be accessed using useDrizzleStorage
      context.db = txn;

      await next();

      delete context.db;

      // Update the indexer's state with cursor.
      await updateState(txn);
    });
  });
});
```

## Chain reorganizations

The indexer needs to be able to rollback state after a chain reorganization.
The behavior described in this section is only relevant for un-finalized blocks.
Finalized blocks don't need special handling since they are, by definition, not going to
be part of a chain reorganization.

The main idea is to create an ["audit table"](https://supabase.com/blog/postgres-audit) with
all changes to the indexer's schema.

The name of the audit table is `__reorg_rollback` and has the following schema.

```txt
+------------+--------------+-----------------------+
| Column     | Type         | Modifiers             |
|------------+--------------+-----------------------|
| n          | integer      |  not null default ... |
| op         | character(1) |  not null             |
| table_name | text         |  not null             |
| cursor     | integer      |  not null             |
| row_id     | text         |                       |
| row_value  | jsonb        |                       |
| indexer_id | text         |  not null             |
+------------+--------------+-----------------------+
```

The data stored in the `row_value` column is specific to each operation (INSERT, DELETE, UPDATE) contains the data needed to revert the operation. Notice that the table's row must be JSON-serializable.

At each block, the plugin registers a trigger for each table managed by the indexer. At the end of the transaction, the trigger inserts data into the audit table.

The audit table is periodically pruned to remove snapshots of data that is now finalized.

### Reverting a block

When a chain reorganization is detected, all operations in the audit table where `cursor` is greater than the new chain's head are reverted in reverse order.

- `op = INSERT`: the row with id `row_id` is deleted from the table.
- `op = DELETE`: the row with id `row_id` is inserted back into the table, with the value stored in `row_value`.
- `op = UPDATE`: the row with id `row_id` is updated in the table, with the value stored in `row_value`.


---
title: Drizzle's plugin - Frequently Asked Questions
description: "Find answers to common questions about using Drizzle with PostgreSQL."
updatedAt: 2025-03-18
---

# Frequently Asked Questions

## General

### Argument of type `PgTableWithColumns` is not assignable to parameter of type `PgTable`.

When structuring your project as monorepo, you may encounter the following error when type checking your project.

```txt
indexers/my-indexer.indexer.ts:172:25 - error TS2345: Argument of type 'PgTableWithColumns<{ name: "my_table"; schema: undefined; columns: { ... }; dialect:...' is not assignable to parameter of type 'PgTable<TableConfig>'.
  The types of '_.config.columns' are incompatible between these types.
    Type '{ ... }' is not assignable to type 'Record<string, PgColumn<ColumnBaseConfig<ColumnDataType, string>, {}, {}>>'.
      Property 'myColumn' is incompatible with index signature.
        Type 'PgColumn<{ name: "my_column"; tableName: "my_table"; dataType: "number"; columnType: "PgBigInt53"; data: number; driverParam: string | number; notNull: false; hasDefault: false; isPrimaryKey: false; ... 5 more ...; generated: undefined; }, {}, {}>' is not assignable to type 'PgColumn<ColumnBaseConfig<ColumnDataType, string>, {}, {}>'.
          The types of 'table._.config.columns' are incompatible between these types.
            Type 'Record<string, import("/my/project/node_modules/.pnpm/drizzle-orm@0.40.1_@types+pg@8.11.10_pg@8.14.1/node_modules/drizzle-orm/pg-core/columns/common").PgColumn<import("/my/project/node_modules/.pnpm/drizzle-orm@0.40.1_@types+pg@8.11.10_pg@8.14.1/node_modules/...' is not assignable to type 'Record<string, import("/my/project/node_modules/.pnpm/drizzle-orm@0.40.1_@types+pg@8.11.11_pg@8.14.1/node_modules/drizzle-orm/pg-core/columns/common").PgColumn<import("/my/project/node_modules/.pnpm/drizzle-orm@0.40.1_@types+pg@8.11.11_pg@8.14.1/node_modules/...'.
              'string' index signatures are incompatible.
                Type 'import("/my/project/node_modules/.pnpm/drizzle-orm@0.40.1_@types+pg@8.11.10_pg@8.14.1/node_modules/drizzle-orm/pg-core/columns/common").PgColumn<import("/my/project/node_modules/.pnpm/drizzle-orm@0.40.1_@types+pg@8.11.10_pg@8.14.1/node_modules/drizzle-orm/col...' is not assignable to type 'import("/my/project/node_modules/.pnpm/drizzle-orm@0.40.1_@types+pg@8.11.11_pg@8.14.1/node_modules/drizzle-orm/pg-core/columns/common").PgColumn<import("/my/project/node_modules/.pnpm/drizzle-orm@0.40.1_@types+pg@8.11.11_pg@8.14.1/node_modules/drizzle-orm/col...'.
                  Property 'config' is protected but type 'Column<T, TRuntimeConfig, TTypeConfig>' is not a class derived from 'Column<T, TRuntimeConfig, TTypeConfig>'.

            await db.insert(myTable).values(rows);
```

This error is caused by different versions of `drizzle-orm`, `pg`, and `@types/pg` being used in different packages in your project.
The solution is to make sure all of them use the same version, delete the `node_modules` folder and reinstall your dependencies.

## Performance

### Why is indexing slower after I add the plugin?

There are many possible reasons for this, but the most common ones are:

- The latency between your indexer and the database is high.
- Your indexer is inserting rows too frequently.

In the first case, consider moving your indexer's deployment closer to the database to improve latency.

In the second case, consider using a bulk insert strategy to reduce the number of individual insert operations.
Usually, this means converting many `db.insert(..)` operations inside a loop into a single `db.insert()` call.

```ts
// Before
for (const event of block.events) {
  const transfer = decodeEvent(event);
  await db.insert(schema.transfers).values(transfer);
}

// After
const transfers = block.events.map((event) => decodeEvent(event));
await db.insert(schema.transfers).values(transfers);
```


---
title: Beacon Chain
description: "Stream Beacon Chain data with Apibara."
diataxis: reference
updatedAt: 2024-12-03
---

# Beacon Chain

```
              ..-#%%%%+..     ....-+#%%%%%%%#*=:...     ...=#%#*:..
             .*@@@@@@@@@@-.:+#@@@@#*+=======+*#%@@@%*=..+@@@@@@@@@%:
            -@@@@@@@@@@@@@@@%+:...             ....-*%@@@@@@@@@@@@@@*.
          ..@@@@@@@@@@@@@#:.                           :%@@@@@@@@@@@@*.
          .#@@@@@@@@@@@+.                               ..-@@@@@@@@@@@-
          :%@@@@@@@@@*..                                   .-%@@@@@@@@#.
          .%@@@@@@@@:                                        .=@@@@@@@#
          .+@@@@@@#.                                           :%@@@@@:
            -@@@@#.                                             .*@@@:
             .*@*                                                .#@+.
            .-@%.        .-#@@%*:.              ..:#@@@#-.       ..#@=.
            :@@.       .-@@@@@@@@+              .*@@@@@@@%:        :@%.
            +@+.      .=@@@@@#.+@#              :@@-:%@@@@@=.      .*@=
          ..@@.       -@@@@@@++@@*              .@@@=+@@@@@@:.      .@%.
          .:@%.       +@@@@@@@@@@:              ..@@@@@@@@@@:.      .%@:
          .-@*.       :@@@@@@@@*.   ..*%%%%#+.    .#@@@@@@@%..      .%@:
          .:@%.      ..:*%@@%*..    .-@@@@@@%.      :*%@@%+::..     .@@.
          ..%@:.     .::::::::..  ..  ..=@=..  ..   ..::::::::.     =@*.
            =@*.     .::::::::..  **...:@@%:...#+.  ..::::::::.    .@@.
            .#@+     ..::::::..   .-*%#=..:=#%*-.    ..::::::..    #@=.
             .*@*..   .......                        .........  ..%@+.
              .=@@-.                                           .=@@-
                .%@@-                                        .+@@+.
                 ..+@@#-...                             ...=%@@=..
                   ..:*@@@*=.                       ..:+#@@%+:.
                      ...-*%@@@@#*+=---::::---=+*#@@@@@#+:...
                              ..-+#%@@@@@@@@@@%#*=:
```

Apibara provides data streams for the Beacon Chain, the Ethereum consensus
layer. Notice that these stream URLs are going to change in the future when DNA
v2 is released.

**Beacon Chain Mainnet**

```txt
https://beaconchain.preview.apibara.org
```

**Beacon Chain Sepolia**

```txt
COMING SOON
```

### Typescript package

Types for the Beacon Chain chain are provided by the `@apibara/beaconchain` package.

```bash [Terminal]
npm install @apibara/beaconchain@next
```


---
title: Beacon Chain filter reference
description: "Beacon Chain: DNA data filter reference guide."
diataxis: reference
updatedAt: 2024-10-22
---

# Beacon Chain filter reference

This page contains reference about the available data filters for Beacon Chain
DNA streams.

### Related pages

- [Beacon Chain block data reference](/docs/v2/networks/beaconchain/data)

## Filter ID

All filters have an associated ID. When the server filters a block, it will
return a list of all filters that matched a piece of data with the data.
You can use this ID to build powerful abstractions in your indexers.

## Filter types

### Root

The root filter object contains a collection of filters.
Notice that providing an empty filter object is an error.

```ts
export type Filter = {
  header?: HeaderFilter;
  transactions: TransactionFilter[];
  blobs: BlobFilter[];
  validators: ValidatorFilter[];
};
```

### Header

The `HeaderFilter` object controls when the block header is returned to the client.

```ts
export type HeaderFilter = "always" | "on_data" | "on_data_or_on_new_block";
```

The values have the following meaning:

- `always`: Always return the header, even if no other filter matches.
- `on_data`: Return the header only if any other filter matches. This is the default value.
- `on_data_or_on_new_block`: Return the header only if any other filter matches. If no other filter matches, return the header only if the block is a new block.

### Transactions

DNA includes decoded transactions submitted to the network.

```ts
export type TransactionFilter = {
  id?: number;
  from?: `0x${string}`;
  to?: `0x${string}`;
  create?: boolean;
  includeBlob?: boolean;
};
```

**Properties**

 - `from`: filter by sender address. If empty, matches any sender address.
 - `to`: filter by receiver address. If empty, matches any receiver address.
 - `create`: filter by whether the transaction is a create transaction.
 - `includeBlob`: also return all blobs included in the transaction.

**Examples**

 - All blobs included in a transaction to a specific contract.

```ts
const filter = [{
  transactions: [{
    to: "0xff00000000000000000000000000000000074248",
    includeBlob: true,
  }],
}];
```

### Blobs

A blob and its content.

```ts
export type BlobFilter = {
  id?: number;
  includeTransaction?: boolean;
};
```

**Properties**

 - `includeTransaction`: also return the transaction that included the blob.

**Examples**

 - All blobs posted to the network together with the transaction that posted them.

```ts
const filter = [{
  blobs: [{
    includeTransaction: true,
  }],
}];
```

### Validators

Validators and their historical balances.

```ts
export type ValidatorStatus =
    | "pending_initialized"
    | "pending_queued"
    | "active_ongoing"
    | "active_exiting"
    | "active_slashed"
    | "exited_unslashed"
    | "exited_slashed"
    | "withdrawal_possible"
    | "withdrawal_done";

export type ValidatorFilter = {
  id?: number;
  validatorIndex?: number;
  status?: ValidatorStatus;
};
```

**Properties**

 - `validatorIndex`: filter by the validator index.
 - `status`: filter by validator status.

**Examples**

 - All validators that exited, both slashed and unlashed.

```ts
const filter = [{
  validators: [{
    status: "exited_unslashed"
  }, {
    status: "exited_slashed"
  }],
}];
```


---
title: Beacon Chain data reference
description: "Beacon Chain: DNA data data reference guide."
diataxis: reference
updatedAt: 2024-10-22
---

# Beacon Chain data reference

This page contains reference about the available data in Beacon Chain DNA streams.

### Related pages

- [Beacon Chain data filter reference](/docs/v2/networks/beaconchain/filter)

## Filter ID

All filters have an associated ID. To help clients correlate filters with data,
the filter ID is included in the `filterIds` field of all data objects.
This field contains the list of _all filter IDs_ that matched a piece of data.

## Nullable fields

**Important**: most fields are nullable to allow evolving the protocol. You should
always assert the presence of a field for critical indexers.

## Scalar types

The `@apibara/beaconchain` package defines the following scalar types:

 - `Address`: a 20-byte Ethereum address, represented as a `0x${string}` type.
 - `B256`: a 32-byte Ethereum value, represented as a `0x${string}` type.
 - `B384`: a 48-byte Ethereum value, represented as a `0x${string}` type.
 - `Bytes`: arbitrary length bytes, represented as a `0x${string}` type.

## Data type

### Block

The root object is the `Block`.

```ts
export type Block = {
  header?: BlockHeader;
  transactions: Transaction[];
  blobs: Blob[];
  validators: Validator[];
};
```

### Header

This is the block header, which contains information about the block.

```ts
export type BlockHeader = {
  slot?: bigint;
  proposerIndex?: number;
  parentRoot?: B256;
  stateRoot?: B256;
  randaoReveal?: Bytes;
  depositCount?: bigint;
  depositRoot?: B256;
  blockHash?: B256;
  graffiti?: B256;
  executionPayload?: ExecutionPayload;
  blobKzgCommitments: B384[];
};

export type ExecutionPayload = {
  parentHash?: B256;
  feeRecipient?: Address;
  stateRoot?: B256;
  receiptsRoot?: B256;
  logsBloom?: Bytes;
  prevRandao?: B256;
  blockNumber?: bigint;
  timestamp?: Date;
};
```

**Properties**

 - `slot`: the slot number.
 - `proposerIndex`: the index of the validator that proposed the block.
 - `parentRoot`: the parent root.
- `stateRoot`: the state root.
- `randaoReveal`: the randao reveal.
- `depositCount`: the number of deposits.
- `depositRoot`: the deposit root.
- `blockHash`: the block hash.
- `graffiti`: the graffiti.
- `executionPayload`: the execution payload.
- `blobKzgCommitments`: the blob kzg commitments.

### Transaction

An EVM transaction.

```ts
export type Transaction = {
  filterIds: number[];
  transactionIndex?: number;
  transactionHash?: B256;
  nonce?: bigint;
  from?: Address;
  to?: Address;
  value?: bigint;
  gasPrice?: bigint;
  gas?: bigint;
  maxFeePerGas?: bigint;
  maxPriorityFeePerGas?: bigint;
  input: Bytes;
  signature?: Signature;
  chainId?: bigint;
  accessList: AccessListItem[],
  transactionType?: bigint;
  maxFeePerBlobGas?: bigint;
  blobVersionedHashes?: B256[];
};

export type Signature = {
  r?: bigint;
  s?: bigint;
  v?: bigint;
  yParity: boolean;
};

export type AccessListItem = {
  address?: Address;
  storageKeys: B256[];
};
```

**Properties**

- `transactionIndex`: the index of the transaction in the block.
- `transactionHash`: the hash of the transaction.
- `nonce`: the nonce of the transaction.
- `from`: the sender of the transaction.
- `to`: the recipient of the transaction. Empty if it's a create transaction.
- `value`: the value of the transaction, in wei.
- `gasPrice`: the gas price of the transaction.
- `gas`: the gas limit of the transaction.
- `maxFeePerGas`: the max fee per gas of the transaction.
- `maxPriorityFeePerGas`: the max priority fee per gas of the transaction.
- `input`: the input data of the transaction.
- `signature`: the signature of the transaction.
- `chainId`: the chain ID of the transaction.
- `accessList`: the access list of the transaction.
- `transactionType`: the transaction type.
- `maxFeePerBlobGas`: the max fee per blob gas of the transaction.
- `blobVersionedHashes`: the hashes of blobs posted by the transaction.
- `transactionStatus`: the status of the transaction.

**Relevant filters**

- `filter.transactions`
- `filter.blobs[].includeTransaction`

### Blob

A blob and its content.

```ts
export type Blob = {
  filterIds: number[];
  blobIndex?: number;
  blob?: Uint8Array;
  kzgCommitment?: B384;
  kzgProof?: B384;
  kzgCommitmentInclusionProof: B256[];
  blobHash?: B256;
  transactionIndex?: number;
  transactionHash?: B256;
};
```

**Properties**

 - `blobIndex`: the index of the blob in the block.
 - `blob`: the blob content.
 - `kzgCommitment`: the blob kzg commitment.
 - `kzgProof`: the blob kzg proof.
 - `kzgCommitmentInclusionProof`: the blob kzg commitment inclusion proof.
 - `blobHash`: the hash of the blob content.
 - `transactionIndex`: the index of the transaction that included the blob.
 - `transactionHash`: the hash of the transaction that included the blob.

**Relevant filters**

 - `filter.blobs`
 - `filter.transactions[].includeBlob`

### Validator

Data about validators.

```ts
export type ValidatorStatus =
    | "pending_initialized"
    | "pending_queued"
    | "active_ongoing"
    | "active_exiting"
    | "active_slashed"
    | "exited_unslashed"
    | "exited_slashed"
    | "withdrawal_possible"
    | "withdrawal_done";

export type Validator = {
  filterIds: number[];
  validatorIndex?: number;
  balance?: bigint;
  status?: ValidatorStatus;
  pubkey?: B384;
  withdrawalCredentials?: B256;
  effectiveBalance?: bigint;
  slashed?: boolean;
  activationEligibilityEpoch?: bigint;
  activationEpoch?: bigint;
  exitEpoch?: bigint;
  withdrawableEpoch?: bigint;
};
```

**Properties**

 - `validatorIndex`: the index of the validator.
 - `balance`: the balance of the validator.
 - `status`: the status of the validator.
 - `pubkey`: the validator's public key.
 - `withdrawalCredentials`: the withdrawal credentials.
 - `effectiveBalance`: the effective balance of the validator.
 - `slashed`: whether the validator is slashed.
 - `activationEligibilityEpoch`: the epoch at which the validator can be activated.
 - `activationEpoch`: the epoch at which the validator was activated.
 - `exitEpoch`: the epoch at which the validator exited.
 - `withdrawableEpoch`: the epoch at which the validator can withdraw.

**Relevant filters**

- `filter.validators`

---
title: Ethereum EVM
description: "Stream Ethereum data with Apibara."
diataxis: reference
updatedAt: 2024-10-22
---

# Ethereum EVM

```

                              @
                             @@@
                           .@@@@@@
                          @@@@@@@@@
                         @@@@@@@@@@@.
                       .@@@@@@@@@@@@@@
                      @@@@@@@@@@@@@@@@@
                     @@@@@@@@@@@@@@@@@@@.
                    @@@@@@@@@@@@@@@@@@@@@@
                  @@@@@@@@@@@@((@@@@@@@@@@@
                 @@@@@@@@@@((((((((@@@@@@@@@.
                @@@@@@@@((((((((((((((@@@@@@@@
              @@@@@@(((((((((((((((((((((@@@@@@
             @@@((((((((((((((((((((((((((((@@@@
            (((((((((((((((((((((((((((((((((((((,
               (((((((((((((((((((((((((((((((^
                  *((((((((((((((((((((((((
                      (((((((((((((((((
              @@@.        (((((((((*       .@@^
                @@@@.        *((       .@@@@@
                 @@@@@@@           ..@@@@@@
                   @@@@@@@@@.   .@@@@@@@@@
                    ^@@@@@@@@@@@@@@@@@@@
                      @@@@@@@@@@@@@@@@@
                       @@@@@@@@@@@@@@
                         @@@@@@@@@@^
                           @@@@@@@
                            @@@@^
                              @

```

Apibara provides data streams for Ethereum mainnet. Notice that these
stream URLs are going to change in the future when DNA v2 is released.

**Ethereum Mainnet**

```txt
https://ethereum.preview.apibara.org
```

**Ethereum Sepolia**

```txt
https://ethereum-sepolia.preview.apibara.org
```

### Typescript package

Types for EVM chains are provided by the `@apibara/evm` package.

```bash [Terminal]
npm install @apibara/evm@next
```


---
title: EVM filter reference
description: "EVM: DNA data filter reference guide."
diataxis: reference
updatedAt: 2024-10-22
---

# EVM filter reference

This page contains reference about the available data filters for EVM DNA
streams.

### Related pages

- [EVM block data reference](/docs/v2/networks/evm/data)

## Filter ID

All filters have an associated ID. When the server filters a block, it will
return a list of all filters that matched a piece of data with the data.
You can use this ID to build powerful abstractions in your indexers.

## Usage with viem

Most types are compatible with [viem](https://viem.sh/).
For example, you can generate log filters with the following code:

```ts
import { encodeEventTopics, parseAbi } from "viem";

const abi = parseAbi([
  "event Transfer(address indexed from, address indexed to, uint256 value)",
]);

const filter = {
  logs: [{
    topics: encodeEventTopics({
      abi,
      eventName: "Transfer",
      args: { from: null, to: null },
    }),
    strict: true,
  }]
}
```

## Filter types

### Root

The root filter object contains a collection of filters.
Notice that providing an empty filter object is an error.

```ts
export type Filter = {
  header?: HeaderFilter;
  logs?: LogFilter[];
  transactions?: TransactionFilter[];
  withdrawals?: WithdrawalFilter[];
};
```

### Header

The `HeaderFilter` object controls when the block header is returned to the client.

```ts
export type HeaderFilter = "always" | "on_data" | "on_data_or_on_new_block";
```

The values have the following meaning:

- `always`: Always return the header, even if no other filter matches.
- `on_data`: Return the header only if any other filter matches. This is the default value.
- `on_data_or_on_new_block`: Return the header only if any other filter matches. If no other filter matches, return the header only if the block is a new block.

## Logs

Logs are the most common type of DNA filters. Use this filter to get the logs and
their associated data like transactions, receipts, and sibling logs.

```ts
export type LogFilter = {
  id?: number;
  address?: `0x${string}`;
  topics?: (`0x${string} | null`)[];
  strict?: boolean;
  transactionStatus?: "succeeded" | "reverted" | "all";
  includeTransaction?: boolean;
  includeReceipt?: boolean;
  includeSiblings?: boolean;
};
```

**Properties**

 - `address`: filter by contract address. If empty, matches any contract
    address.
 - `topics`: filter by topic. Use `null` to match _any_ value.
 - `strict`: return logs whose topics length matches the filter. By default, the
    filter does a prefix match on the topics.
 - `transactionStatus`: return logs emitted by transactions with the provided
    status. Defaults to `succeeded`.
 - `includeTransaction`: also return the transaction that emitted the log.
 - `includeReceipt`: also return the receipt of the transaction that emitted the
    log.
 - `includeSiblings`: also return all other logs emitted by the same transaction
    that emitted the matched log.

**Examples**

 - All logs in a block emitted by successful transactions.

```ts
const filter = {
  logs: [{}],
};
```

 - All `Transfer` events emitted by successful transactions. Notice that this will
   match logs from ERC-20, ERC-721, and other contracts that emit `Transfer`.

```ts
const filter = {
  logs: [{
    topics: ["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"],
  }],
};
```

 - All `Transfer` events that follow the ERC-721 standard. Notice that this will
   not match logs from ERC-20 since the number of indexed parameters is different.

```ts
const filter = {
  logs: [{
    topics: [
        "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef",
        null, // from
        null, // to
        null, // tokenId
    ],
    strict: true,
  }],
};
```

 - All logs emitted by `CONTRACT_A` OR `CONTRACT_B`.
 
```ts
const filter = {
  logs: [{
    address: CONTRACT_A,
  }, {
    address: CONTRACT_B,
  }],
};
```

## Transactions

Request Ethereum transactions.

```ts
export type TransactionFilter = {
  id?: number;
  from?: `0x${string}`;
  to?: `0x${string}`;
  create?: true,
  transactionStatus?: "succeeded" | "reverted" | "all";
  includeReceipt?: boolean;
  includeLogs?: boolean;
};
```

**Properties**

 - `from`: filter by sender address. If empty, matches any sender address.
 - `to`: filter by receiver address. If empty, matches any receiver address.
 - `create`: filter by whether the transaction is a create transaction.
 - `transactionStatus`: return transactions with the provided status. Defaults to
    `succeeded`.
 - `includeReceipt`: also return the receipt of the transaction.
 - `includeLogs`: also return the logs emitted by the transaction.

**Examples**

 - All transactions in a block.
 
```ts
const filter = {
  transactions: [{}],
};
```

 - All transactions from `0xAB...`.

```ts
const filter = {
  transactions: [{
    from: "0xAB...",
  }],
};
```

 - All create transactions.

```ts
const filter = {
  transactions: [{
    create: true,
  }],
};
```

## Withdrawals

Request Ethereum withdrawals.

```ts
export type WithdrawalFilter = {
  id?: number;
  validatorIndex?: number;
  address?: string;
};
```

**Properties**

 - `validatorIndex`: filter by validator's index. If empty, matches any
    validator's index.
 - `address`: filter by withdrawal address. If empty, matches any withdrawal
    address.

**Examples**

 - All withdrawals

```ts
const filter = {
  withdrawals: [{}],
};
```

 - All withdrawals from validator with index `1234`.

```ts
const filter = {
  withdrawals: [{
    validatorIndex: 1234,
  }],
};
```

 - All withdrawals from validators with index `1234` OR `7890`.

```ts
const filter = {
  withdrawals: [{
    validatorIndex: 1234,
  }, {
    validatorIndex: 7890,
  }],
};
```

 - All withdrawals to address `0xAB...`.

```ts
const filter = {
  withdrawals: [{
    address: "0xAB...",
  }],
};
```


---
title: EVM data reference
description: "EVM: DNA data data reference guide."
diataxis: reference
updatedAt: 2024-10-22
---

# EVM data reference

This page contains reference about the available data in EVM DNA streams.

### Related pages

- [EVM data filter reference](/docs/v2/networks/evm/filter)

## Filter ID

All filters have an associated ID. To help clients correlate filters with data,
the filter ID is included in the `filterIds` field of all data objects.
This field contains the list of _all filter IDs_ that matched a piece of data.

## Nullable fields

**Important**: most fields are nullable to allow evolving the protocol. You should
always assert the presence of a field for critical indexers.

## Scalar types

The `@apibara/evm` package defines the following scalar types:

 - `Address`: a 20-byte Ethereum address, represented as a `0x${string}` type.
 - `B256`: a 32-byte Ethereum value, represented as a `0x${string}` type.
 - `Bytes`: arbitrary length bytes, represented as a `0x${string}` type.

## Usage with viem

Most types are compatible with [viem](https://viem.sh/).
For example, you can decode logs with the following code:

```ts
import type { B256 } from "@apibara/evm";
import { decodeEventLog, parseAbi } from "viem";

const abi = parseAbi([
  "event Transfer(address indexed from, address indexed to, uint256 value)",
]);

// Somewhere in your indexer...
for (const log of logs) {
  const { args, eventName } = decodeEventLog({
    abi,
    topics: log.topics as [B256, ...B256[]],
    data: log.data,
  });
}
```

## Data type

### Block

The root object is the `Block`.

```ts
export type Block = {
  header?: BlockHeader;
  logs: Log[];
  transactions: Transaction[];
  receipts: TransactionReceipt[];
  withdrawals: Withdrawal[];
};
```

### Header

This is the block header, which contains information about the block.

```ts
export type Bloom = Bytes;

export type BlockHeader = {
  blockNumber?: bigint;
  blockHash?: B256,
  parentBlockHash?: B256;
  unclesHash?: B256;
  miner?: Address;
  stateRoot?: B256;
  transactionsRoot?: B256;
  receiptsRoot?: B256;
  logsBloom?: Bloom;
  difficulty?: bigint;
  gasLimit?: bigint;
  gasUsed?: bigint;
  timestamp?: Date;
  extraData?: Bytes;
  mixHash?: B256;
  nonce?: bigint;
  baseFeePerGas?: bigint;
  withdrawalsRoot?: B256;
  totalDifficulty?: bigint;
  blobGasUsed?: bigint;
  excessBlobGas?: bigint;
  parentBeaconBlockRoot?: B256;
};
```

**Properties**

 - `blockNumber`: the block number.
 - `blockHash`: the block hash.
 - `parentBlockHash`: the block hash of the parent block.
 - `unclesHash`: the block hash of the uncles.
 - `miner`: the address of the miner.
 - `stateRoot`: the state root.
 - `transactionsRoot`: the transactions root.
 - `receiptsRoot`: the receipts root.
 - `logsBloom`: the logs bloom.
 - `difficulty`: the block difficulty.
 - `gasLimit`: the block gas limit.
 - `gasUsed`: the gas used by transactions in the block.
 - `timestamp`: the block timestamp.
 - `extraData`: extra bytes data picked by the miner.
 - `mixHash`: the mix hash.
 - `nonce`: the nonce.
 - `baseFeePerGas`: the base fee per gas.
 - `withdrawalsRoot`: the withdrawals root.
 - `totalDifficulty`: the total difficulty.
 - `blobGasUsed`: the gas used by transactions posting blob data in the block.
 - `excessBlobGas`: the excess blob gas.
 - `parentBeaconBlockRoot`: the parent beacon block root.

### Log

An EVM log. It comes together with the essential information about the transaction
that emitted the log.

```ts
export type Log = {
  filterIds: number[];
  address?: Address;
  topics: B256[];
  data: Bytes;
  logIndex: number;
  transactionIndex: number;
  transactionHash: B256;
  transactionStatus: "succeeded" | "reverted";
};
```

**Properties**

 - `address`: the address of the contract that emitted the log.
 - `topics`: the topics of the log.
 - `data`: the data of the log.
 - `logIndex`: the index of the log in the block.
 - `transactionIndex`: the index of the transaction that emitted the log.
 - `transactionHash`: the hash of the transaction that emitted the log.
 - `transactionStatus`: the status of the transaction that emitted the log.

**Relevant filters**

 - `filter.logs`
 - `filter.logs[].includeSiblings`
 - `filter.transactions[].includeLogs`

### Transaction

An EVM transaction.

```ts
export type Transaction = {
  filterIds: number[];
  transactionIndex?: number;
  transactionHash?: B256;
  nonce?: bigint;
  from?: Address;
  to?: Address;
  value?: bigint;
  gasPrice?: bigint;
  gas?: bigint;
  maxFeePerGas?: bigint;
  maxPriorityFeePerGas?: bigint;
  input: Bytes;
  signature?: Signature;
  chainId?: bigint;
  accessList: AccessListItem[],
  transactionType?: bigint;
  maxFeePerBlobGas?: bigint;
  blobVersionedHashes?: B256[];
  transactionStatus: "succeeded" | "reverted";
};

export type Signature = {
  r?: bigint;
  s?: bigint;
  v?: bigint;
  yParity: boolean;
};

export type AccessListItem = {
  address?: Address;
  storageKeys: B256[];
};
```

**Properties**

- `transactionIndex`: the index of the transaction in the block.
- `transactionHash`: the hash of the transaction.
- `nonce`: the nonce of the transaction.
- `from`: the sender of the transaction.
- `to`: the recipient of the transaction. Empty if it's a create transaction.
- `value`: the value of the transaction, in wei.
- `gasPrice`: the gas price of the transaction.
- `gas`: the gas limit of the transaction.
- `maxFeePerGas`: the max fee per gas of the transaction.
- `maxPriorityFeePerGas`: the max priority fee per gas of the transaction.
- `input`: the input data of the transaction.
- `signature`: the signature of the transaction.
- `chainId`: the chain ID of the transaction.
- `accessList`: the access list of the transaction.
- `transactionType`: the transaction type.
- `maxFeePerBlobGas`: the max fee per blob gas of the transaction.
- `blobVersionedHashes`: the hashes of blobs posted by the transaction.
- `transactionStatus`: the status of the transaction.

**Relevant filters**

- `filter.transactions`
- `filter.logs[].includeTransaction`

### Transaction Receipt

Information about the transaction's execution.

```ts
export type TransactionReceipt = {
  filterIds: number[];
  transactionIndex?: number;
  transactionHash?: B256;
  cumulativeGasUsed?: bigint;
  gasUsed?: bigint;
  effectiveGasPrice?: bigint;
  from?: Address;
  to?: Address;
  contractAddress?: Address;
  logsBloom?: Bloom;
  transactionType?: bigint;
  blobGasUsed?: bigint;
  blobGasPrice?: bigint;
  transactionStatus: "succeeded" | "reverted";
};
```

**Properties**

- `transactionIndex`: the transaction index in the block.
- `transactionHash`: the hash of the transaction.
- `cumulativeGasUsed`: the cumulative gas used by the transactions.
- `gasUsed`: the gas used by the transaction.
- `effectiveGasPrice`: the effective gas price of the transaction.
- `from`: the sender of the transaction.
- `to`: the recipient of the transaction. Empty if it's a create transaction.
- `contractAddress`: the address of the contract created by the transaction.
- `logsBloom`: the logs bloom of the transaction.
- `transactionType`: the transaction type.
- `blobGasUsed`: the gas used by the transaction posting blob data.
- `blobGasPrice`: the gas price of the transaction posting blob data.
- `transactionStatus`: the status of the transaction.

**Relevant filters**

- `filter.transactions[].includeReceipt`
- `filter.logs[].includeReceipt`

### Withdrawal

A withdrawal from the Ethereum network.

```ts
export type Withdrawal = {
  filterIds: number[];
  withdrawalIndex?: number;
  index?: bigint;
  validatorIndex?: number;
  address?: Address;
  amount?: bigint;
};
```

**Properties**

 - `withdrawalIndex`: the index of the withdrawal in the block.
 - `index`: the global index of the withdrawal.
 - `validatorIndex`: the index of the validator that created the withdrawal.
 - `address`: the destination address of the withdrawal.
 - `amount`: the amount of the withdrawal, in wei.

**Relevant filters**

 - `filter.withdrawals`


---
title: Starknet
description: "Stream Starknet data with Apibara."
diataxis: reference
updatedAt: 2024-10-22
---

# Starknet

```

                                                   ,//(//,.
                     (@.                    .#@@@@@@@@@@@@@@@@@#.
                    (@@&.                (@@@@@@@@@@@@@@@@@@@@@@@@&.
                /%@@@@@@@@@%.         ,@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@,
                    .@@#            *@@@@@@@@@@@@@@@@@@@@@@@&&&&&&&@@@%
                     *%           ,@@@@@@@@@@@@@@@@@@@@#((((((((((((,
                                .@@@@@@@@@@@@@@@@@@@%((((((((((((,
                              .@@@@@@@@@@@@@@@@@@@&((((((((((((.
                            .@@@@@@@@@@@@@@@@@@@@%(((((((((((.
                          *@@@@@@@@@@@@@@@@@@@@&(((((((((((,
                      .%@@@@@@@@@@@@@@@@@@@@@@#((((((((((*
        @@@@@&&&@@@@@@@@@@@@@@@@@@@@@@@@@@@@#((((((((((/
         %@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@#(((((((((((.
          ,&@@@@@@@@@@@@@@@@@@@@@@@@@@@#((((((((((((.
            ,#@@@@@@@@@@@@@@@@@@@@@&#((((((((((((*        .,**.
              ./(#%@@@@@@@@@@@&#((((((((((((((/         *(((((((/
                  *(((((((((((((((((((((((/.            (((((((((
                      .*/(((((((((((/*.                  /((((((.



```

Apibara provides data streams for all Starknet networks. Notice that these
stream URLs are going to change in the future when DNA v2 is released.

**Starknet Mainnet**

```txt
https://starknet.preview.apibara.org
```

**Starknet Sepolia**

```txt
https://starknet-sepolia.preview.apibara.org
```

## Starknet appchains

You can ingest data from Starknet appchains and serve it using our open-source
DNA service. Please get in touch with the team if you'd like a managed solution.


---
title: Starknet filter reference
description: "Starknet: DNA data filter reference guide."
diataxis: reference
updatedAt: 2024-10-23
---

# Starknet filter reference

This page contains reference about the available data filters for Starknet DNA
streams.

### Related pages

- [Starknet block data reference](/docs/v2/networks/starknet/data)

## Filter ID

All filters have an associated ID. When the server filters a block, it will
return a list of all filters that matched a piece of data with the data.
You can use this ID to build powerful abstractions in your indexers.

## Field elements

Apibara represents Starknet field elements as hex-encode strings.

```ts
export type FieldElement = `0x${string}`;
```

## Filter types

### Root

The root filter object contains a collection of filters.
Notice that providing an empty filter object is an error.

```ts
export type Filter = {
  header?: HeaderFilter;
  transactions?: TransactionFilter[];
  events?: EventFilter[];
  messages?: MessageToL1Filter[];
  storageDiffs?: StorageDiffFilter[];
  contractChanges?: ContractChangeFilter[];
  nonceUpdates?: NonceUpdateFilter[];
};
```

### Header

The `HeaderFilter` object controls when the block header is returned to the client.

```ts
export type HeaderFilter = "always" | "on_data" | "on_data_or_on_new_block";
```

The values have the following meaning:

- `always`: Always return the header, even if no other filter matches.
- `on_data`: Return the header only if any other filter matches. This is the default value.
- `on_data_or_on_new_block`: Return the header only if any other filter matches. If no other filter matches, return the header only if the block is a new block.

### Events

Events are the most common filter used by Apibara users. You can filter by smart contract or event selector.

```ts
export type EventFilter = {
  id?: number;
  address?: FieldElement;
  keys?: (FieldElement | null)[];
  strict?: boolean;
  transactionStatus?: "succeeded" | "reverted" | "all";
  includeTransaction?: boolean;
  includeReceipt?: boolean;
  includeMessages?: boolean;
  includeSiblings?: boolean;
};
```

**Properties**

 - `address`: filter by contract address. If empty, matches any contract
    address.
 - `keys`: filter by keys. Use `null` to match _any_ value. The server will
    filter based only the first four elements of the array.
 - `strict`: return events whose keys length matches the filter. By default, the
    filter does a prefix match on the keys.
 - `transactionStatus`: return events emitted by transactions with the provided
    status. Defaults to `succeeded`.
 - `includeTransaction`: also return the transaction that emitted the event.
 - `includeReceipt`: also return the receipt of the transaction that emitted the
    event.
 - `includeMessages`: also return all messages to L1 sent by the transaction that
    emitted the event.
 - `includeSiblings`: also return all other events emitted by the same transaction
    that emitted the matched event.

**Examples**

- All events from a specific smart contract.

```ts
const filter = {
  events: [{ address: MY_CONTRACT }],
};
```

- Multiple events from the same smart contract.

```ts
const filter = {
  events: [
    {
      address: MY_CONTRACT,
      keys: [APPROVE_SELECTOR],
    },
    {
      address: MY_CONTRACT,
      keys: [TRANSFER_SELECTOR],
    },
  ],
};
```

- Multiple events from different smart contracts.

```ts
const filter = {
  events: [
    {
      address: CONTRACT_A,
      keys: [TRANSFER_SELECTOR],
    },
    {
      address: CONTRACT_B,
      keys: [TRANSFER_SELECTOR],
      includeReceipt: false,
    },
    {
      address: CONTRACT_C,
      keys: [TRANSFER_SELECTOR],
    },
  ],
};
```

### Transactions

Transactions on Starknet can be of different type (invoke, declare contract, deploy contract or account, handle L1 message). Clients can request all transactions or filter by transaction type.

```ts
export type InvokeTransactionV0Filter = {
  _tag: "invokeV0";
  invokeV0: {};
};

export type InvokeTransactionV1Filter = {
  _tag: "invokeV1";
  invokeV1: {};
};

export type InvokeTransactionV3Filter = {
  _tag: "invokeV3";
  invokeV3: {};
};

export type DeployTransactionFilter = {
  _tag: "deploy";
  deploy: {};
};

export type DeclareV0TransactionFilter = {
  _tag: "declareV0";
  declareV0: {};
};

export type DeclareV1TransactionFilter = {
  _tag: "declareV1";
  declareV1: {};
};

export type DeclareV2TransactionFilter = {
  _tag: "declareV2";
  declareV2: {};
};

export type DeclareV3TransactionFilter = {
  _tag: "declareV3";
  declareV3: {};
};

export type L1HandlerTransactionFilter = {
  _tag: "l1Handler";
  l1Handler: {};
};

export type DeployAccountV1TransactionFilter = {
  _tag: "deployAccountV1";
  deployAccountV1: {};
};

export type DeployAccountV3TransactionFilter = {
  _tag: "deployAccountV3";
  deployAccountV3: {};
};

export type TransactionFilter = {
  id?: number;
  transactionStatus?: "succeeded" | "reverted" | "all";
  includeReceipt?: boolean;
  includeMessages?: boolean;
  includeEvents?: boolean;
  transactionType?: 
      | InvokeTransactionV0Filter
      | InvokeTransactionV1Filter
      | InvokeTransactionV3Filter
      | DeployTransactionFilter
      | DeclareV0TransactionFilter
      | DeclareV1TransactionFilter
      | DeclareV2TransactionFilter
      | DeclareV3TransactionFilter
      | L1HandlerTransactionFilter
      | DeployAccountV1TransactionFilter
      | DeployAccountV3TransactionFilter;
};
```

**Properties**

 - `transactionStatus`: return transactions with the provided status. Defaults to
    `succeeded`.
 - `includeReceipt`: also return the receipt of the transaction.
 - `includeMessages`: also return the messages to L1 sent by the transaction.
 - `includeEvents`: also return the events emitted by the transaction.
 - `transactionType`: filter by transaction type. 

**Examples**

- Request all transactions in a block. Notice the empty transaction filter object, this filter will match _any_ transaction.

```ts
const filter = { transactions: [{}] };
```

- Request all transactions of a specific type, e.g. deploy account. In this case we specify the `deployAccountV3` variant.

```ts
const filter = {
  transactions: [{
    transactionType: { _tag: "deployAccountV3", deployAccountV3: {} }
  }]
};
```

### Messages

Filter messages from L1 to Starknet.

```ts
export type MessageToL1Filter = {
  id?: number;
  fromAddress?: FieldElement;
  toAddress?: FieldElement;
  transactionStatus?: "succeeded" | "reverted" | "all";
  includeTransaction?: boolean;
  includeReceipt?: boolean;
  includeEvents?: boolean;
};
```

**Properties**

 - `fromAddress`: filter by sender address. If empty, matches any sender address.
 - `toAddress`: filter by receiver address. If empty, matches any receiver address.
 - `transactionStatus`: return messages with the provided status. Defaults to
    `succeeded`.
 - `includeTransaction`: also return the transaction that sent the message.
 - `includeReceipt`: also return the receipt of the transaction that sent the
    message.
 - `includeEvents`: also return the events emitted by the transaction that sent 
    the message.

### Storage diff

Request changes to the storage of one or more contracts.

```ts
export type StorageDiffFilter = {
  id?: number;
  contractAddress?: FieldElement;
};
```

**Properties**

 - `contractAddress`: filter by contract address. If empty, matches any contract
    address.

### Contract change

Request changes to the declared or deployed contracts.

```ts
export type DeclaredClassFilter = {
  _tag: "declaredClass";
  declaredClass: {};
};

export type ReplacedClassFilter = {
  _tag: "replacedClass";
  replacedClass: {};
};

export type DeployedContractFilter = {
  _tag: "deployedContract";
  deployedContract: {};
};

export type ContractChangeFilter = {
  id?: number;
  change?: DeclaredClassFilter | ReplacedClassFilter | DeployedContractFilter;
};
```

**Properties**

 - `change`: filter by change type.
   + `declaredClass`: receive declared classes.
   + `replacedClass`: receive replaced classes.
   + `deployedContract`: receive deployed contracts.

### Nonce update

Request changes to the nonce of one or more contracts.

```ts
export type NonceUpdateFilter = {
  id?: number;
  contractAddress?: FieldElement;
};
```

**Properties**

 - `contractAddress`: filter by contract address. If empty, matches any contract.


---
title: Starknet data reference
description: "Starknet: DNA data data reference guide."
diataxis: reference
updatedAt: 2024-10-23
---

# Starknet data reference

This page contains reference about the available data in Starknet DNA streams.

### Related pages

- [Starknet data filter reference](/docs/v2/networks/starknet/filter)

## Filter ID

All filters have an associated ID. To help clients correlate filters with data,
the filter ID is included in the `filterIds` field of all data objects.
This field contains the list of _all filter IDs_ that matched a piece of data.

## Nullable fields

**Important**: most fields are nullable to allow evolving the protocol. You should
always assert the presence of a field for critical indexers.

## Field elements

Apibara represents Starknet field elements as hex-encode strings.

```ts
export type FieldElement = `0x${string}`;
```

## Data types

### Block

The root object is the `Block`.

```ts
export type Block = {
  header?: BlockHeader;
  transactions: Transaction[];
  receipts: TransactionReceipt[];
  events: Event[];
  messages: MessageToL1[];
  storageDiffs: StorageDiff[];
  contractChanges: ContractChange[];
  nonceUpdates: NonceUpdate[];
};
```

### Block header

This is the block header, which contains information about the block.

```ts
export type BlockHeader = {
  blockHash?: FieldElement;
  parentBlockHash?: FieldElement;
  blockNumber?: bigint;
  sequencerAddress?: FieldElement;
  newRoot?: FieldElement;
  timestamp?: Date;
  starknetVersion?: string;
  l1GasPrice?: ResourcePrice;
  l1DataGasPrice?: ResourcePrice;
  l1DataAvailabilityMode?: "blob" | "calldata";
};

export type ResourcePrice = {
  priceInFri?: FieldElement;
  priceInWei?: FieldElement;
};
```

**Properties**

 - `blockHash`: the block hash.
 - `parentBlockHash`: the block hash of the parent block.
 - `blockNumber`: the block number.
 - `sequencerAddress`: the sequencer address.
 - `newRoot`: the new state root.
 - `timestamp`: the block timestamp.
 - `starknetVersion`: the Starknet version.
 - `l1GasPrice`: the L1 gas price.
 - `l1DataGasPrice`: the L1 data gas price.
 - `l1DataAvailabilityMode`: the L1 data availability mode.

 - `priceInFri`: the price of L1 gas in the block, in units of fri (10^-18 $STRK).
 - `priceInWei`: the price of L1 gas in the block, in units of wei (10^-18 $ETH).

### Event

An event is emitted by a transaction.

```ts
export type Event = {
  filterIds: number[];
  address?: FieldElement;
  keys: FieldElement[];
  data: FieldElement[];
  eventIndex?: number;
  transactionIndex?: number;
  transactionHash?: FieldElement;
  transactionStatus?: "succeeded" | "reverted";
};
```

**Properties**

 - `address`: the address of the contract that emitted the event.
 - `keys`: the keys of the event.
 - `data`: the data of the event.
 - `eventIndex`: the index of the event in the block.
 - `transactionIndex`: the index of the transaction that emitted the event.
 - `transactionHash`: the hash of the transaction that emitted the event.
 - `transactionStatus`: the status of the transaction that emitted the event.

**Relevant filters**

- `filter.events`
- `filter.transactions[].includeEvents`
- `filter.events[].includeSiblings`
- `filter.messages[].includeEvents`

### Transaction

Starknet has different types of transactions, all of them are grouped together
in the `Transaction` type. Common transaction information is accessible in the
`meta` field.

```ts
export type TransactionMeta = {
  transactionIndex?: number;
  transactionHash?: FieldElement;
  transactionStatus?: "succeeded" | "reverted";
};

export type Transaction = {
  filterIds: number[];
  meta?: TransactionMeta;
  transaction?: 
      | InvokeTransactionV0
      | InvokeTransactionV1
      | InvokeTransactionV3
      | L1HandlerTransaction
      | DeployTransaction
      | DeclareTransactionV0
      | DeclareTransactionV1
      | DeclareTransactionV2
      | DeclareTransactionV3
      | DeployAccountTransactionV1
      | DeployAccountTransactionV3;
};
```

**Properties**

 - `meta`: transaction metadata.
 - `transaction`: the transaction type.

 - `meta.transactionIndex`: the index of the transaction in the block.
 - `meta.transactionHash`: the hash of the transaction.
 - `meta.transactionStatus`: the status of the transaction.

**Relevant filters**

 - `filter.transactions`
 - `filter.events[].includeTransaction`
 - `filter.messages[].includeTransaction`

```ts
export type InvokeTransactionV0 = {
  _tag: "invokeV0";
  invokeV0: {
    maxFee?: FieldElement;
    signature: FieldElement[];
    contractAddress?: FieldElement;
    entryPointSelector?: FieldElement;
    calldata: FieldElement[];
  };
}
```

**Properties**

 - `maxFee`: the maximum fee for the transaction.
 - `signature`: the signature of the transaction.
 - `contractAddress`: the address of the contract that will receive the call.
 - `entryPointSelector`: the selector of the function that will be called.
 - `calldata`: the calldata of the transaction.

```ts
export type InvokeTransactionV1 = {
  _tag: "invokeV1";
  invokeV1: {
    senderAddress?: FieldElement;
    calldata: FieldElement[];
    maxFee?: FieldElement;
    signature: FieldElement[];
    nonce?: FieldElement;
  };
};
```

**Properties**

 - `senderAddress`: the address of the account that will send the transaction.
 - `calldata`: the calldata of the transaction.
 - `maxFee`: the maximum fee for the transaction.
 - `signature`: the signature of the transaction.
 - `nonce`: the nonce of the transaction.

```ts
export type ResourceBounds = {
  maxAmount?: bigint;
  maxPricePerUnit?: bigint;
}

export type ResourceBoundsMapping = {
  l1Gas?: ResourceBounds;
  l2Gas?: ResourceBounds;
};

export type InvokeTransactionV3 = {
  _tag: "invokeV3";
  invokeV3: {
    senderAddress?: FieldElement;
    calldata: FieldElement[];
    signature: FieldElement[];
    nonce?: FieldElement;
    resourceBounds?: ResourceBoundsMapping;
    tip?: bigint;
    paymasterData: FieldElement[];
    accountDeploymentData: FieldElement[];
    nonceDataAvailabilityMode?: "l1" | "l2";
    feeDataAvailabilityMode?: "l1" | "l2";
  };
};
```

**Properties**

 - `senderAddress`: the address of the account that will send the transaction.
 - `calldata`: the calldata of the transaction.
 - `signature`: the signature of the transaction.
 - `nonce`: the nonce of the transaction.
 - `resourceBounds`: the resource bounds of the transaction.
 - `tip`: the tip of the transaction.
 - `paymasterData`: the paymaster data of the transaction.
 - `accountDeploymentData`: the account deployment data of the transaction.
 - `nonceDataAvailabilityMode`: the nonce data availability mode of the transaction.
 - `feeDataAvailabilityMode`: the fee data availability mode of the transaction.

```ts
export type L1HandlerTransaction = {
  _tag: "l1Handler";
  l1Handler: {
    contractAddress?: FieldElement;
    entryPointSelector?: FieldElement;
    calldata: FieldElement[];
    nonce?: bigint;
  };
}
```

**Properties**

 - `contractAddress`: the address of the contract that will receive the call.
 - `entryPointSelector`: the selector of the function that will be called.
 - `calldata`: the calldata of the transaction.
 - `nonce`: the nonce of the transaction.

```ts
export type DeployTransaction = {
  _tag: "deploy";
  deploy: {
    contractAddressSalt?: FieldElement;
    constructorCalldata: FieldElement[];
    classHash?: FieldElement;
  };
}
```

**Properties**

 - `contractAddressSalt`: the salt used to compute the contract address.
 - `constructorCalldata`: the calldata used to initialize the contract.
 - `classHash`: the class hash of the contract.

```ts
export type DeclareTransactionV0 = {
  _tag: "declareV0";
  declareV0: {
    senderAddress?: FieldElement;
    maxFee?: FieldElement;
    signature: FieldElement[];
    classHash?: FieldElement;
  };
}
```

**Properties**

 - `senderAddress`: the address of the account that will send the transaction.
 - `maxFee`: the maximum fee for the transaction.
 - `signature`: the signature of the transaction.
 - `classHash`: the class hash of the contract.

```ts
export type DeclareTransactionV1 = {
  _tag: "declareV1";
  declareV1: {
    senderAddress?: FieldElement;
    maxFee?: FieldElement;
    signature: FieldElement[];
    classHash?: FieldElement;
    nonce?: FieldElement;
  };
}
```

**Properties**

 - `senderAddress`: the address of the account that will send the transaction.
 - `maxFee`: the maximum fee for the transaction.
 - `signature`: the signature of the transaction.
 - `classHash`: the class hash of the contract.
 - `nonce`: the nonce of the transaction.

```ts
export type DeclareTransactionV2 = {
  _tag: "declareV2";
  declareV2: {
    senderAddress?: FieldElement;
    maxFee?: FieldElement;
    signature: FieldElement[];
    classHash?: FieldElement;
    nonce?: FieldElement;
    compiledClassHash?: FieldElement;
  };
}
```

**Properties**

 - `senderAddress`: the address of the account that will send the transaction.
 - `maxFee`: the maximum fee for the transaction.
 - `signature`: the signature of the transaction.
 - `classHash`: the class hash of the contract.
 - `nonce`: the nonce of the transaction.
 - `compiledClassHash`: the compiled class hash of the contract.

```ts
export type ResourceBounds = {
  maxAmount?: bigint;
  maxPricePerUnit?: bigint;
}

export type ResourceBoundsMapping = {
  l1Gas?: ResourceBounds;
  l2Gas?: ResourceBounds;
};

export type DeclareTransactionV3 = {
  _tag: "declareV3";
  declareV3: {
    senderAddress?: FieldElement;
    maxFee?: FieldElement;
    signature: FieldElement[];
    classHash?: FieldElement;
    nonce?: FieldElement;
    compiledClassHash?: FieldElement;
    resourceBounds?: ResourceBoundsMapping;
    tip?: bigint;
    paymasterData: FieldElement[];
    accountDeploymentData: FieldElement[];
    nonceDataAvailabilityMode?: "l1" | "l2";
    feeDataAvailabilityMode?: "l1" | "l2";
  };
}
```

**Properties**

 - `senderAddress`: the address of the account that will send the transaction.
 - `maxFee`: the maximum fee for the transaction.
 - `signature`: the signature of the transaction.
 - `classHash`: the class hash of the contract.
 - `nonce`: the nonce of the transaction.
 - `compiledClassHash`: the compiled class hash of the contract.
 - `resourceBounds`: the resource bounds of the transaction.
 - `tip`: the tip of the transaction.
 - `paymasterData`: the paymaster data of the transaction.
 - `accountDeploymentData`: the account deployment data of the transaction.
 - `nonceDataAvailabilityMode`: the nonce data availability mode of the transaction.
 - `feeDataAvailabilityMode`: the fee data availability mode of the transaction.

```ts
export type DeployAccountTransactionV1 = {
  _tag: "deployAccountV1";
  deployAccountV1: {
    maxFee?: FieldElement;
    signature: FieldElement[];
    nonce?: FieldElement;
    contractAddressSalt?: FieldElement;
    constructorCalldata: FieldElement[];
    classHash?: FieldElement;
  };
}
```

**Properties**

 - `maxFee`: the maximum fee for the transaction.
 - `signature`: the signature of the transaction.
 - `nonce`: the nonce of the transaction.
 - `contractAddressSalt`: the salt used to compute the contract address.
 - `constructorCalldata`: the calldata used to initialize the contract.
 - `classHash`: the class hash of the contract.

```ts
export type DeployAccountTransactionV3 = {
  _tag: "deployAccountV3";
  deployAccountV3: {
    signature: FieldElement[];
    nonce?: FieldElement;
    contractAddressSalt?: FieldElement;
    constructorCalldata: FieldElement[];n
    classHash?: FieldElement;
    resourceBounds?: ResourceBoundsMapping;
    tip?: bigint;
    paymasterData: FieldElement[];
    nonceDataAvailabilityMode?: "l1" | "l2";
    feeDataAvailabilityMode?: "l1" | "l2";
  };
};
```

**Properties**

 - `signature`: the signature of the transaction.
 - `nonce`: the nonce of the transaction.
 - `contractAddressSalt`: the salt used to compute the contract address.
 - `constructorCalldata`: the calldata used to initialize the contract.
 - `classHash`: the class hash of the contract
 - `resourceBounds`: the resource bounds of the transaction.
 - `tip`: the tip of the transaction.
 - `paymasterData`: the paymaster data of the transaction.
 - `nonceDataAvailabilityMode`: the nonce data availability mode of the transaction.
 - `feeDataAvailabilityMode`: the fee data availability mode of the transaction.

### Transaction receipt

The receipt of a transaction contains information about the execution of the transaction.

```ts
export type TransactionReceipt = {
  filterIds: number[];
  meta?: TransactionReceiptMeta;
  receipt?:
    | InvokeTransactionReceipt
    | L1HandlerTransactionReceipt
    | DeclareTransactionReceipt
    | DeployTransactionReceipt
    | DeployAccountTransactionReceipt;
};

export type TransactionReceiptMeta = {
  transactionIndex?: number;
  transactionHash?: FieldElement;
  actualFee?: FeePayment;
  executionResources?: ExecutionResources;
  executionResult?: ExecutionSucceeded | ExecutionReverted;
};

export type InvokeTransactionReceipt = {
  _tag: "invoke";
  invoke: {}
};

export type L1HandlerTransactionReceipt = {
  _tag: "l1Handler";
  l1Handler: {
    messageHash?: Uint8Array;
  };
};

export type DeclareTransactionReceipt = {
  _tag: "declare";
  declare: {};
};

export type DeployTransactionReceipt = {
  _tag: "deploy";
  deploy: {
    contractAddress?: FieldElement;
  };
};

export type DeployAccountTransactionReceipt = {
  _tag: "deployAccount";
  deployAccount: {
    contractAddress?: FieldElement;
  };
};

export type ExecutionSucceeded = {
  _tag: "succeeded";
  succeeded: {};
};

export type ExecutionReverted = {
  _tag: "reverted";
  reverted: {
    reason: string;
  };
};

export type FeePayment = {
  amount?: FieldElement;
  unit?: "wei" | "strk";
};
```

**Relevant filters**

 - `filter.transactions[].includeReceipt`
 - `filter.events[].includeReceipt`
 - `filter.messages[].includeReceipt`

### Message to L1

A message to L1 is sent by a transaction.

```ts
export type MessageToL1 = {
  filterIds: number[];
  fromAddress?: FieldElement;
  toAddress?: FieldElement;
  payload: FieldElement[];
  messageIndex?: number;
  transactionIndex?: number;
  transactionHash?: FieldElement;
  transactionStatus?: "succeeded" | "reverted";
};
```

**Properties**

 - `fromAddress`: the address of the contract that sent the message.
 - `toAddress`: the address of the contract that received the message.
 - `payload`: the payload of the message.
 - `messageIndex`: the index of the message in the block.
 - `transactionIndex`: the index of the transaction that sent the message.
 - `transactionHash`: the hash of the transaction that sent the message.
 - `transactionStatus`: the status of the transaction that sent the message.

**Relevant filters**

 - `filter.messages`
 - `filter.transactions[].includeMessages`
 - `filter.events[].includeMessages`

### Storage diff

A storage diff is a change to the storage of a contract.

```ts
export type StorageDiff = {
  filterIds: number[];
  contractAddress?: FieldElement;
  storageEntries: StorageEntry[];
};

export type StorageEntry = {
  key?: FieldElement;
  value?: FieldElement;
};
```

**Properties**

 - `contractAddress`: the contract whose storage changed.
 - `storageEntries`: the storage entries that changed.

 - `key`: the key of the storage entry that changed.
 - `value`: the new value of the storage entry that changed.

**Relevant filters**

 - `filter.storageDiffs`

### Contract change

A change in the declared or deployed contracts.

```ts
export type ContractChange = {
  filterIds: number[];
  change?: DeclaredClass | ReplacedClass | DeployedContract;
};

export type DeclaredClass = {
  _tag: "declaredClass";
  declaredClass: {
    classHash?: FieldElement;
    compiledClassHash?: FieldElement;
  };
};

export type ReplacedClass = {
  _tag: "replacedClass";
  replacedClass: {
    contractAddress?: FieldElement;
    classHash?: FieldElement;
  };
};

export type DeployedContract = {
  _tag: "deployedContract";
  deployedContract: {
    contractAddress?: FieldElement;
    classHash?: FieldElement;
  };
};
```

**Relevant filters**

 - `filter.contractChanges`

### Nonce update

A change in the nonce of a contract.

```ts
export type NonceUpdate = {
  filterIds: number[];
  contractAddress?: FieldElement;
  nonce?: FieldElement;
};
```

**Properties**

 - `contractAddress`: the address of the contract whose nonce changed.
 - `nonce`: the new nonce of the contract.

**Relevant filters**

 - `filter.nonceUpdates`

---
title: Upgrading from v1
description: "This page explains how to upgrade from DNA v1 to DNA v2."
diataxis: how-to
updatedAt: 2024-11-06
---

# Upgrading from v1

This page contains a list of changes between DNA v1 and DNA v2.

## @apibara/starknet package

This package now works in combination with `@apibara/protocol` to provide a DNA stream
that automatically encodes and decodes the Protobuf data. This means tha field elements
are automatically converted to `0x${string}` values.

Notice that the data stream is now unary.

```js
import { createClient } from "@apibara/protocol";
import { Filter, StarknetStream } from "@apibara/starknet";

const client = createClient(StarknetStream, process.env.STREAM_URL);

const filter = Filter.make({
  events: [{
    address:
      "0x049d36570d4e46f48e99674bd3fcc84644ddd6b96f7c741b1562b82f9e004dc7",
  }],
});

const request = StarknetStream.Request.make({
  filter: [filter],
  finality: "accepted",
  startingCursor: {
    orderKey: 800_000n,
  },
});

for await (const message of client.streamData(request)) {
  switch (message._tag) {
    case "data": {
      break;
    }
    case "invalidate": {
      break;
    }
    default: {
      break;
    }
  }
}
```

### Reconnecting on error

The client now doesn't automatically reconnect on error. This is because the reconnection step
is very delicate and depends on your indexer's implementation.
The recommended approach is to wrap your indexer's main loop in a `try/catch` block.

```ts
import { createClient, type ClientError, type Status } from "@apibara/protocol";
import { Filter, StarknetStream } from "@apibara/starknet";

const client = createClient(StarknetStream, process.env.STREAM_URL);

const filter = Filter.make({
  events: [{
    address:
      "0x049d36570d4e46f48e99674bd3fcc84644ddd6b96f7c741b1562b82f9e004dc7",
  }],
});

while (true) {
  try {
    const startingCursor = await loadCursorFromDatabase();
    const request = StarknetStream.Request.make({
      filter: [filter],
      finality: "accepted",
      startingCursor,
    });
  
    for await (const message of client.streamData(request)) {
    }
  } catch (err) {
    if (err instanceof ClientError) {
      // It's a gRPC error.
      if (err.status !== Status.INTERNAL) {
        // NON-INTERNAL errors are not recoverable.
        throw err;
      }
  
      // INTERNAL errors are caused by a disconnection.
      // Sleep and reconnect.
      await new Promise(r => setTimeout(r, 2_000));
    }
  }
}
```

## Filter

### Header

 - The `header` field is now an enum. See the [dedicated section](/docs/v2/networks/starknet/filter#header)
in the filter documentation for more information.

### Events

 - `fromAddress` is now `address`.
 - The `keys` field accepts `null` values to match any key at that position.
 - The `data` field was removed.
 - Use `transactionStatus: "all"` instead of `includeReverted` to include reverted transactions.
 - `includeReceipt` and `includeTransaction` are now `false` by default.

### Transactions

 - Now you can only filter by transaction type.
 - We will add transaction-specific filters in the future.
 - Use `transactionStatus: "all"` instead of `includeReverted` to include reverted transactions.
 - `includeReceipt` is now `false` by default.

### Messages

 - Can now filter by `fromAddress` and `toAddress`.
 - Use `transactionStatus: "all"` instead of `includeReverted` to include reverted transactions.
 - `includeReceipt` and `includeTransaction` are now `false` by default.

### State Update
 
 - State update has been split into separate filters for storage diffs, contract
   changes, and nonce updates.
 - Declared and deployed contracts, declared classes, and replaced classes are now
   a single `contractChanges` filter.

## Block data

 - Block data has been _"flattened"_. Use the `*Index` field to access related data.
   For example, the following code iterates over all events and looks up their
   transactions.

```js
for (const event of block.events) {
  const transaction = block.transactions.find(tx => tx.transactionIndex === event.transactionIndex);
}
```

### Events

 - `fromAddress` is now `address`.
 - `index` is now `eventIndex`.
 - Events now include `transactionIndex`, `transactionHash`, and `transactionStatus`.

### Transactions

 - `TransactionMeta` now includes `transactionIndex`, `transactionHash`, and `transactionStatus`.
 - The transaction type is now an enum using the `_tag` field as discriminator.
 - For other minor changes, see the [transaction documentation](/docs/v2/networks/starknet/data#transaction).

### Receipts

 - Transaction receipts are now transaction-specific.
 - For other minor changes, see the [receipts documentation](/docs/v2/networks/starknet/data#transaction-receipt).

### Messages

 - `index` is now `messageIndex`.
 - Messages now include `transactionIndex`, `transactionHash`, and `transactionStatus`.


---
title: DNA protocol & architecture
description: "Learn about the low-level DNA streaming protocol to access onchain data."
diataxis: explanation
updatedAt: 2024-09-20
---

# DNA protocol & architecture

This section describes the internals of DNA v2.

 - [Wire protocol](/docs/v2/dna/protocol): describes the gRPC streaming
   protocol. This page is useful if you're connecting directly to the stream or
   are adding support for a new programming language.
 - [Architecture](/docs/v2/dna/architecture): describes the high-level components of DNA v2.
 - [Adding a new chain](/docs/v2/dna/add-new-chain): describes what you
   need to do to bring DNA to a new chain. It digs deeper into anything
   chain-specific like storage and filters.



---
title: DNA v2 architecture
description: "Discover how DNA achieves best-in-class performance for indexing onchain data."
diataxis: explanation
updatedAt: 2024-09-20
---

# DNA v2 architecture

This page describes in detail the architecture of DNA v2.

At a high-level, the goals for DNA v2 are:

- serve onchain data through a protocol that's optimized for building indexers.
- provide a scalable and cost-efficient way to access onchain data.
- decouple compute from storage.

This is achieved by building a _cloud native_ service that ingests onchain data
from an archive node and stores it into Object Storage (for example Amazon S3,
Cloudflare R2). Data is served by stateless workers that read and filter data
from Object Storage before sending it to the indexers. The diagram below shows
all the high-level components that make a production deployment of DNA v2.
Communication between components is done through etcd.

```txt
                 ┌─────────────────────────────────────────────┐                 
                 │                Archive Node                 │░                
                 └─────────────────────────────────────────────┘░                
                  ░░░░░░░░░░░░░░░░░░░░░░│░░░░░░░░░░░░░░░░░░░░░░░░                
                                        │                                        
                                        │                                        
  ╔═ DNA Cluster ═══════════════════════╬══════════════════════════════════════╗ 
  ║                                     │                                      ║░
  ║ ┌──────┐                            ▼                            ┌──────┐  ║░
  ║ │      │     ┌─────────────────────────────────────────────┐     │      │  ║░
  ║ │      │     │                                             │     │      │  ║░
  ║ │      │◀────│              Ingestion Service              │────▶│      │  ║░
  ║ │      │     │                                             │     │      │  ║░
  ║ │      │     └─────────────────────────────────────────────┘     │      │  ║░
  ║ │      │     ┌─────────────────────────────────────────────┐     │      │  ║░
  ║ │      │     │                                             │     │      │  ║░
  ║ │      │◀────│             Compaction Service              │────▶│      │  ║░
  ║ │      │     │                                             │     │      │  ║░
  ║ │      │     └─────────────────────────────────────────────┘     │      │  ║░
  ║ │  S3  │     ┌─────────────────────────────────────────────┐     │ etcd │  ║░
  ║ │      │     │                                             │     │      │  ║░
  ║ │      │◀────│               Pruning Service               │────▶│      │  ║░
  ║ │      │     │                                             │     │      │  ║░
  ║ │      │     └─────────────────────────────────────────────┘     │      │  ║░
  ║ │      │     ┌───────────────────────────────────────────┐       │      │  ║░
  ║ │      │     │┌──────────────────────────────────────────┴┐      │      │  ║░
  ║ │      │     ││┌──────────────────────────────────────────┴┐     │      │  ║░
  ║ │      │     │││                                           │     │      │  ║░
  ║ │      │     │││                  Stream                   │     │      │  ║░
  ║ │      │◀────┤││                                           ├────▶│      │  ║░
  ║ │      │     │││                  Service                  │     │      │  ║░
  ║ └──────┘     └┤│                                           │     └──────┘  ║░
  ║               └┤                                           │               ║░
  ║                └───────────────────────────────────────────┘               ║░
  ║                                                                            ║░
  ╚════════════════════════════════════════════════════════════════════════════╝░
   ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
```

## DNA service

The DNA service is comprised of several components:

- ingestion service: listens for new blocks on the network and stores them into
  Object Storage.
- compaction service: combines multiple blocks together into _segments_.
  Segments are grouped by data type (like logs, transactions, and receipts).
- pruner service: removes blocks that have been compacted to reduce storage cost.
- stream service: receives streaming requests from clients (indexers) and serves
  onchain data by filtering objects stored on S3.

### Ingestion service

The ingestion service fetches blocks from the network and stores them into
Object Storage. This service is the only chain-specific service in DNA, all
other components work on generic data-structures.

Serving onchain data requires serving a high-volume of data filtered by a
relatively small number of columns. When designing DNA, we took a few decisions
to make this process as efficient as possible:

- data is stored as pre-serialized protobuf messages to avoid wasting CPU
  cycles serializing the same data over and over again.
- filtering is entirely done using indices to reduce reads.
- joins (for example include logs' transactions) are also achieved with indices.

The ingestion service is responsible for creating this data and indices.
Data is grouped into _blocks_. Blocks are comprised of _fragments_, that
is groups of related data. All fragments have an unique numerical id used
to identify them. There are four different types of fragments:

- index: a collection of indices, the fragment id is `0`.
  Indices are grouped by the fragment they index.
- join: a collection of join indices, the fragment id is `254`.
  Join indices are also grouped by the source fragment index.
- header: the block header, the fragment id is `1`. Header are stored as
  pre-serialized protobuf messages.
- body: the chain-specific block data, grouped by fragment id.

Note that we call block number + hash a _cursor_ since it uniquely identifies a
block in the chain.

```txt
 ╔═ Block ══════════════════════════════════════════════════════════════╗ 
 ║ ┌─ Index ──────────────────────────────────────────────────────────┐ ║░
 ║ │ ┌─ Fragment 0 ─────────────────────────────────────────────────┐ │ ║░
 ║ │ │┌────────────────────────────────────────────────────────────┐│ │ ║░
 ║ │ ││                          Index 0                           ││ │ ║░
 ║ │ │├────────────────────────────────────────────────────────────┤│ │ ║░
 ║ │ ││                          Index 1                           ││ │ ║░
 ║ │ │├────────────────────────────────────────────────────────────┤│ │ ║░
 ║ │ │                                                              │ │ ║░
 ║ │ │├────────────────────────────────────────────────────────────┤│ │ ║░
 ║ │ ││                          Index N                           ││ │ ║░
 ║ │ │└────────────────────────────────────────────────────────────┘│ │ ║░
 ║ │ └──────────────────────────────────────────────────────────────┘ │ ║░
 ║ └──────────────────────────────────────────────────────────────────┘ ║░
 ║ ┌─ Join ───────────────────────────────────────────────────────────┐ ║░
 ║ │ ┌─ Fragment 0 ─────────────────────────────────────────────────┐ │ ║░
 ║ │ │┌────────────────────────────────────────────────────────────┐│ │ ║░
 ║ │ ││                         Fragment 1                         ││ │ ║░
 ║ │ │├────────────────────────────────────────────────────────────┤│ │ ║░
 ║ │ ││                         Fragment 2                         ││ │ ║░
 ║ │ │└────────────────────────────────────────────────────────────┘│ │ ║░
 ║ │ └──────────────────────────────────────────────────────────────┘ │ ║░
 ║ └──────────────────────────────────────────────────────────────────┘ ║░
 ║ ┌─ Body ───────────────────────────────────────────────────────────┐ ║░
 ║ │ ┌──────────────────────────────────────────────────────────────┐ │ ║░
 ║ │ │                                                              │ │ ║░
 ║ │ │                           Fragment 0                         │ │ ║░
 ║ │ │                                                              │ │ ║░
 ║ │ └──────────────────────────────────────────────────────────────┘ │ ║░
 ║ │ ┌──────────────────────────────────────────────────────────────┐ │ ║░
 ║ │ │                                                              │ │ ║░
 ║ │ │                           Fragment 1                         │ │ ║░
 ║ │ │                                                              │ │ ║░
 ║ │ └──────────────────────────────────────────────────────────────┘ │ ║░
 ║ └──────────────────────────────────────────────────────────────────┘ ║░
 ╚══════════════════════════════════════════════════════════════════════╝░
 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
```

On supported networks, the ingestion service is also responsible for
periodically refreshing the mempool (pending) block data and uploading it into
Object Storage. This works exactly as for all other blocks.

The ingestion service tracks the canonical chain and uploads it to Object Storage.
This data is used by the stream service to track online and offline chain
reorganizations.

The ingestion service stores its data on etcd. Stream services subscribe to
etcd updates to receive notifications about new blocks ingested and other
changes to the chain (for example changes to the finalized block).

Finally, the ingestion service is _fault tolerant_. When the ingestion service
starts, it acquires a distributed lock from etcd to ensure only one instance is
running at the same time. If running multiple deployments of DNA, all other
instances will wait for the lock to be released (following a service restart or
crash) and will try to take over the ingestion.

### Compaction service

The compaction service groups together data from several blocks (usually 100 or
1000) into _segments_. Segments only contain data for one fragment type (for example 
headers, indices, and transactions).
In other words, the compaction service groups `N` blocks into `M` segments.

Only data that has been finalized is compacted into segments.

The compaction service also creates block-level indices called _groups_. Groups
combine indices from multiple blocks/segments to quickly look up which blocks
contain specific data. This type of index is very useful to increase performance
on sparse datasets.

```txt
 ╔═ Index Segment ═══════════════════════╗         ╔═ Transaction Segment ═════════════════╗ 
 ║ ┌─ Block ───────────────────────────┐ ║░        ║ ┌─ Block ───────────────────────────┐ ║░
 ║ │ ┌─ Fragment 0 ──────────────────┐ │ ║░        ║ │ ┌───────────────────────────────┐ │ ║░
 ║ │ │┌─────────────────────────────┐│ │ ║░        ║ │ │                               │ │ ║░
 ║ │ ││           Index 0           ││ │ ║░        ║ │ │                               │ │ ║░
 ║ │ │├─────────────────────────────┤│ │ ║░        ║ │ │           Fragment 2          │ │ ║░
 ║ │ ││           Index 1           ││ │ ║░        ║ │ │                               │ │ ║░
 ║ │ │├─────────────────────────────┤│ │ ║░        ║ │ │                               │ │ ║░
 ║ │ │                               │ │ ║░        ║ │ └───────────────────────────────┘ │ ║░
 ║ │ │├─────────────────────────────┤│ │ ║░        ║ └───────────────────────────────────┘ ║░
 ║ │ ││           Index N           ││ │ ║░        ║ ┌─ Block ───────────────────────────┐ ║░
 ║ │ │└─────────────────────────────┘│ │ ║░        ║ │ ┌───────────────────────────────┐ │ ║░
 ║ │ └───────────────────────────────┘ │ ║░        ║ │ │                               │ │ ║░
 ║ └───────────────────────────────────┘ ║░        ║ │ │                               │ │ ║░
 ║ ┌─ Block ───────────────────────────┐ ║░        ║ │ │           Fragment 2          │ │ ║░
 ║ │ ┌─ Fragment 0 ──────────────────┐ │ ║░        ║ │ │                               │ │ ║░
 ║ │ │┌─────────────────────────────┐│ │ ║░        ║ │ │                               │ │ ║░
 ║ │ ││           Index 0           ││ │ ║░        ║ │ └───────────────────────────────┘ │ ║░
 ║ │ │├─────────────────────────────┤│ │ ║░        ║ └───────────────────────────────────┘ ║░
 ║ │ ││           Index 1           ││ │ ║░        ║ ┌─ Block ───────────────────────────┐ ║░
 ║ │ │├─────────────────────────────┤│ │ ║░        ║ │ ┌───────────────────────────────┐ │ ║░
 ║ │ │                               │ │ ║░        ║ │ │                               │ │ ║░
 ║ │ │├─────────────────────────────┤│ │ ║░        ║ │ │           Fragment 2          │ │ ║░
 ║ │ ││           Index N           ││ │ ║░        ║ │ │                               │ │ ║░
 ║ │ │└─────────────────────────────┘│ │ ║░        ║ │ │                               │ │ ║░
 ║ │ └───────────────────────────────┘ │ ║░        ║ │ └───────────────────────────────┘ │ ║░
 ║ └───────────────────────────────────┘ ║░        ║ └───────────────────────────────────┘ ║░
 ╚═══════════════════════════════════════╝░        ╚═══════════════════════════════════════╝░
  ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░         ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
```

### Pruner service

The pruner service cleans up block data that has been included in segments.
This is done to reduce the storage used by DNA.

### Object hierarchy

We now have all elements to understand the objects uploaded to Object Storage by the
ingestion service.
If you run DNA pointing it to your bucket, you can eventually see a folder structure
that looks like the following.

```txt
my-chain
├── blocks
│   ├── 000020908017
│   │   └── 0xc137607affd53bd9e857af372429762f77eaff0fe32f0e49224e9fc0e439118d
│   │   ├── pending-0
│   │   ├── pending-1
│   │   └── pending-2
│   ├── 000020908018
│   │   └── ... same as above
│   └── 000020908019
│       └── ... same as above
├── chain
│   ├── recent
│   ├── z-000020906000
│   ├── z-000020907000
│   └── z-000020908000
├── groups
│   └── 000020905000
│       └── index
└── segments
    ├── 000020906000
    │   ├── header
    │   ├── index
    │   ├── join
    │   ├── log
    │   ├── receipt
    │   └── transaction
    ├── 000020907000
    │   └── ... same as above
    └── 000020908000
        └── ... same as above
```

### Stream service

The stream service is responsible for serving data to clients. The raw onchain
data stored in Object Storage is filtered by the stream service before being
sent over the network, this results in lower egress fees compared to solutions
that filter data on the client.

Upon receiving a stream request, the service validates and compiles the request
into a _query_. A query is simply a list of index lookup requests that are
applied to each block.

The stream service loops then keeps repeating the following steps:

- check if it should send a new block of data or inform the client of a chain
  reorganization.
- load the indices from the segment or the block and use them to compute what
  data to send the client.
- load the pre-serialized protobuf messages and copy them to the output stream.

One critical aspect of the stream service is how it loads blocks and segments.
Reading from Object Storage has virtually unlimited throughput, but also high
latency. The service is also very likely to access data closer to the chain's
tip more frequently, and we should cache Object Storage requests to avoid
unnecessarily increase our cloud spending.

We achieve all of this (and more!) by using an hybrid cache that stores
frequently accessed data in memory and _on disk_. This may come as a surprise
since isn't the point of DNA to avoid expensive disks and rely on cheap Object
Storage? The reasons this design still makes sense are multiple:

- we can use cheaper and higher performance temporary NVMe disks attached
  directly to our server.
- we can quickly scale horizontally the stream service without re-indexing all
  data.
- we can use disks that are much smaller than the full chain's data. The cache
  dynamically stores the most frequently accessed data. Unused or rarely used
  data lives on Object Storage.

The following table, inspired [by the table in this article by
Vantage](https://www.vantage.sh/blog/ebs-vs-nvme-pricing-performance), shows the
difference in performance and price between an AWS EC2 instance using
(temporary) NVMe disks and two using EBS (one with a general purpose `gp3`
volume, and one with higher performance `io1` volume). All prices as of April
2024, US East, 1 year reserved with no upfront payment.

| Metric                | EBS (gp3)   | EBS (io1)   | NVMe              |
| --------------------- | ----------- | ----------- | ----------------- |
| Instance Type         | r6i.8xlarge | r6i.8xlarge | i4i.8xlarge       |
| vCPU                  | 32          | 32          | 32                |
| Memory (GiB)          | 256         | 256         | 245               |
| Network (Gibps)       | 12.50       | 12.50       | 18.75             |
| Storage (GiB)         | 7500        | 7500        | 2x3750            |
| IOPS (read/write)     | 16,000      | 40,000      | 800,000 / 440,000 |
| Cost - Compute ($/mo) | 973         | 973         | 1,300             |
| Cost - Storage ($/mo) | 665         | 3,537       | 0                 |
| Cost - Total ($/mo)   | 1,638       | 4,510       | 1,300             |

Notice how the NVMe instance has 30-50x the IOPS per dollar. This price
difference means that Apibara users benefit from lower costs and/or higher
performance.



---
title: DNA wire protocol
description: "DNA is a protocol built on top of gRPC to stream onchain data."
diataxis: explanation
updatedAt: 2024-10-10
---

# DNA wire protocol

## `Cursor` message

Before explaining the DNA protocol in more detail, we're going to discuss the
`Cursor` message type. This type is used by all methods discussed later and
plays a central role in how DNA works.

DNA models a blockchain as a sequence of blocks. The distance of a block from
the first block in the chain (the genesis block) is known as chain height. The
genesis block has height `0`. Ideally, a blockchain should always build a block
on top of the most recent block, but that's not always the case. For this
reason, a block's height isn't enough to uniquely identify a block in the
blockchain. A _chain reorganization_ is when a chain produces blocks that are
not building on top of the most recent block. As we will see later, the DNA
protocol detects and handles chain reorganizations.
A block that can't be part of a chain reorganization is _finalized_.

DNA uses a _cursor_ to uniquely identify blocks on the chain. A cursor contains
two fields:

 - `order_key`: the block's height.
 - `unique_key`: the block's unique identifier. Depending on the chain, it's
   the block hash or state root.


## `Status` method

The `Status` method is used to retrieve the state of the DNA server. The request
is an empty message. The response has the following fields:

 - `last_ingested`: returns the last block ingested by the server. This is the most
    recent block available for streaming.
 - `finalized`: the most recent finalized block.
 - `starting`: the first available block. Usually this is the genesis block,
   but DNA server operators can prune older nodes to save on storage space.

## `StreamData` method

The `StreamData` method is used to start a DNA stream. It accepts a `StreamDataRequest`
message and returns an infinite stream of `StreamDataResponse` messages.

### Request

The request message is used to configure the stream. All fields except `filter`
are optional.

 - `starting_cursor`: resume the stream from the provided cursor. The first
   block received in the stream will be the block following the provided
   cursor. If no cursor is provided, the stream will start from the genesis
   block. Notice that since `starting_cursor` is a cursor, the DNA server can
   detect if that block has been part of a chain's reorganization while the
   indexer was offline.
 - `finality`: the stream contains data with at least the specified finality.
   Possible values are _finalized_ (only receive finalized data), _accepted_
   (receive finalized and non-finalized blocks), and _pending_ (receive
   finalized, non-finalized, and pending blocks).
 - `filter`: a non-empty list of chain-specific data filters.
 - `heartbeat_interval`: the stream will send an heartbeat message if there are
   no messages for the specified amount of time. This is useful to detect if
   the stream hangs. Value must be between 10 and 60 seconds.

### Response

Once the server validates and accepts the request, it starts streaming data.
Each stream message can be one of the following message types:

 - `data`: receive data about a block.
 - `invalidate`: the specified blocks don't belong to the canonical chain
   anymore because they were part of a chain reorganization.
 - `finalize`: the most recent finalized block moved forward.
 - `heartbeat`: an heartbeat message.
 - `system_message`: used to send messages from the server to the client.

#### `Data` message

Contains the requested data for a single block. All data messages cursors are
monotonically increasing, unless an `Invalidate` message is received.

The message contains the following fields:

 - `cursor`: the cursor of the block before this message. If the client
   reconnects using this cursor, the first message will be the same as this
   message.
 - `end_cursor`: this block's cursor. Reconnecting to the stream using this
   cursor will resume the stream.
 - `finality`: finality status of this block.
 - `data`: a list of encoded block data.

Notice how the `data` field is a _list of block data_. This sounds
counter-intuitive since the `Data` message contains data about a _single
block_. The reason is that, as we've seen in the _"Request"_ section,
the client can specify a list of filters. The `data` field has the same
length as the request's `filters` field.
In most cases, the client specifies a single filter and receives a single block
of data. For advanced use cases (like tracking contracts deployed by a
factory), the client uses multiple filters to have parallel streams of data
synced on the block number.

#### `Invalidate` message

This message warns the client about a chain reorganization. It contains the
following fields:

 - `cursor`: the new chain's head. All previously received messages where the
   `end_cursor.order_key` was greater than (`>`) this message
   `cursor.order_key` should be considered invalid/recalled.
 - `removed`: a list of cursors that used to belong to the canonical chain.

#### `Finalize` message

This message contains a single `cursor` field with the cursor of the most
recent finalized block. All data at or before this block can't be part of a
chain reorganization.

This message is useful to prune old data.

#### `Heartbeat` message

This message is sent at regular intervals once the stream reaches the chain's head.

Clients can detect if the stream hang by adding a timeout to the stream's _receive_
method.

#### `SytemMessage` message

This message is used by the server to send out-of-band messages to the client.
It contains text messages such as data usage, warnings about reaching the free
quota, or information about upcoming system upgrades.

## protobuf definition

This section contains the protobuf definition used by the DNA server and clients.
If you're implementing a new SDK for DNA, you can use this as the starting point.

```proto
syntax = "proto3";

package dna.v2.stream;

import "google/protobuf/duration.proto";

service DnaStream {
  // Stream data from the server.
  rpc StreamData(StreamDataRequest) returns (stream StreamDataResponse);
  // Get DNA server status.
  rpc Status(StatusRequest) returns (StatusResponse);
}

// A cursor over the stream content.
message Cursor {
  // Key used for ordering messages in the stream.
  //
  // This is usually the block or slot number.
  uint64 order_key = 1;
  // Key used to discriminate branches in the stream.
  //
  // This is usually the hash of the block.
  bytes unique_key = 2;
}

// Request for the `Status` method.
message StatusRequest {}

// Response for the `Status` method.
message StatusResponse {
  // The current head of the chain.
  Cursor current_head = 1;
  // The last cursor that was ingested by the node.
  Cursor last_ingested = 2;
  // The finalized block.
  Cursor finalized = 3;
  // The first block available.
  Cursor starting = 4;
}

// Request data to be streamed.
message StreamDataRequest {
  // Cursor to start streaming from.
  //
  // If not specified, starts from the genesis block.
  // Use the data's message `end_cursor` field to resume streaming.
  optional Cursor starting_cursor = 1;
  // Return data with the specified finality.
  //
  // If not specified, defaults to `DATA_FINALITY_ACCEPTED`.
  optional DataFinality finality = 2;
  // Filters used to generate data.
  repeated bytes filter = 3;
  // Heartbeat interval.
  //
  // Value must be between 10 and 60 seconds.
  // If not specified, defaults to 30 seconds.
  optional google.protobuf.Duration heartbeat_interval = 4;
}

// Contains a piece of streamed data.
message StreamDataResponse {
  oneof message {
    Data data = 1;
    Invalidate invalidate = 2;
    Finalize finalize = 3;
    Heartbeat heartbeat = 4;
    SystemMessage system_message = 5;
  }
}

// Invalidate data after the given cursor.
message Invalidate {
  // The cursor of the new chain's head.
  //
  // All data after this cursor should be considered invalid.
  Cursor cursor = 1;
  // List of blocks that were removed from the chain.
  repeated Cursor removed = 2;
}

// Move the finalized block forward.
message Finalize {
  // The cursor of the new finalized block.
  //
  // All data before this cursor cannot be invalidated.
  Cursor cursor = 1;
}

// A single block of data.
//
// If the request specified multiple filters, the `data` field will contain the
// data for each filter in the same order as the filters were specified in the
// request.
// If no data is available for a filter, the corresponding data field will be
// empty.
message Data {
  // Cursor that generated this block of data.
  optional Cursor cursor = 1;
  // Block cursor. Use this cursor to resume the stream.
  Cursor end_cursor = 2;
  // The finality status of the block.
  DataFinality finality = 3;
  // The block data.
  //
  // This message contains chain-specific data serialized using protobuf.
  repeated bytes data = 4;
}

// Sent to clients to check if stream is still connected.
message Heartbeat {}

// Message from the server to the client.
message SystemMessage {
  oneof output {
    // Output to stdout.
    string stdout = 1;
    // Output to stderr.
    string stderr = 2;
  }
}

// Data finality.
enum DataFinality {
  DATA_FINALITY_UNKNOWN = 0;
  // Data was received, but is not part of the canonical chain yet.
  DATA_FINALITY_PENDING = 1;
  // Data is now part of the canonical chain, but could still be invalidated.
  DATA_FINALITY_ACCEPTED = 2;
  // Data is finalized and cannot be invalidated.
  DATA_FINALITY_FINALIZED = 3;
}
```


---
title: Adding a new chain
description: "Learn how to bring DNA to your chain, giving developers access to the best indexing platform on the market."
diataxis: how-to
updatedAt: 2024-09-22
---

# Adding a new chain

This page explains how to add support for a new chain to the DNA protocol. It's recommended that you're familiar with the high-level [DNA architecture](/docs/v2/dna/architecture) and the [DNA streaming protocol](/docs/v2/dna/protocol) before reading this page.

## Overview

Adding a new chain is relatively straightforward. Most of the code you need to write is describing the type of data stored on your chain.
The guide is split in the following sections:

 - **gRPC Protocol**: describes how to augment the gRPC protocol with filters and data types specific to the new chain.
 - **Storage**: describes how data is stored on disk and S3. 
 - **Data filtering**: describes how to filter data based on the client's request.

## gRPC Protocol

The first step is to define the root `Filter` and `Block` protobuf messages.

There are a few hard requirements on the messages:

 - The `header` field of the block must have tag `1`.
 - All other fields can have any tag.
 - Add one message type for each chain's resource (transactions, receipts, logs, etc.).
 - Each resource must have a `filter_ids` field with tag `1`.
 - Add a `Filter.id: uint32` property. Indexers use this to know which filters matched a specific piece of data and is used to populate the `filter_ids` field.

The following items are optional:

 - Add an option to the `Filter` to request all block headers. Users use this to debug their indexer.
 - Think how users are going to use the data. For example, developers often access the transaction's hash of a log, for this reason we include the transaction hash in the `Log` message.
 - Avoid excessive nesting of messages.

## Storage

The goal of the ingestion service is to fetch data from the chain (using the chain's RPC protocol), preprocess and index it, and then store it into the object storage.

DNA stores block data as pre-serialized protobuf messages. This is done to send data to clients by 
copying bytes directly, without expensive serialization and deserialization.

Since DNA doesn't know about the chain, it needs a way to filter data without scanning the entire block.
This is done with _indices_.

The chain-specific ingestion service is responsible for creating these indices. The next section goes into
detail how indices work, the important part is that:

 - Indices are grouped by the type of data they index (for example transactions, logs, and traces).
 - For each type of data, there can be multiple indices.
 - Indices point to one or more pre-serialized protobuf messages.

## Data filtering

As mentioned in the previous section, the DNA server uses indices to lookup data without scanning the entire block.
This is done by compiling the protobuf filter sent by the client into a special representation.
This `Filter` specifies:

 - What resource to filter (for example transactions, logs, and traces).
 - The list of conditions to match.

A _condition_ is a tuple with the filter id and the lookup key.
