Batch Plugin

Combine multiple requests into a single batch and receive their responses together, reducing the overhead of separate requests.

Setup

Set up batching on both the server and the client. The server plugin handles incoming batch requests, and the client plugin groups outgoing requests into batches.

import { BatchHandlerPlugin } from '@orpc/server/plugins'

const handler = new RPCHandler(router, {
  plugins: [new BatchHandlerPlugin()],
})

import { BatchLinkPlugin } from '@orpc/client/plugins'

const link = new RPCLink({
  url: '/rpc',
  plugins: [
    new BatchLinkPlugin({
      groups: [
        {
          condition: () => true,
          context: {},
        },
      ],
    }),
  ],
})

Both plugins limit how many requests a batch holds with maxSize, and each side defaults to DEFAULT_BATCH_MAX_SIZE (10) independently. The client splits a larger group into several batches, while the server rejects a larger batch with a 413 for every request in it, so keep the client limit at or below the server’s.

const handler = new RPCHandler(router, {
  plugins: [
    new BatchHandlerPlugin({
      maxSize: 20,
    }),
  ],
})

const link = new RPCLink({
  url: '/rpc',
  plugins: [
    new BatchLinkPlugin({
      maxSize: 20,
      groups: [
        {
          condition: () => true,
          context: {},
        },
      ],
    }),
  ],
})

Response Modes

In the default buffered mode, the server starts the batch response once every procedure has responded, so it can carry their headers. The response streams only if one of them returned an AsyncIteratorObject or a ReadableStream. If your client cannot read streamed responses, keep these procedures out of the batch with filter.

In streaming mode, the server starts the response once the first procedure responds and sends each other response as soon as it is ready, so a slow procedure does not hold back the rest:

const link = new RPCLink({
  url: '/rpc',
  plugins: [
    new BatchLinkPlugin({
      mode: 'streaming',
      groups: [
        {
          condition: () => true,
          context: {},
        },
      ],
    }),
  ],
})

Keep-Alive Timer

A streaming batch response may stay idle for long periods. The server plugin can send keep-alive frames to keep the connection alive.

const handler = new RPCHandler(router, {
  plugins: [
    new BatchHandlerPlugin({
      /**
       * Keep-alive frames sent periodically while the stream is idle.
       *
       * `false` disables it, an object customizes it.
       *
       * @default true
       */
      keepAlive: {
        /**
         * Interval (in milliseconds) between keep-alive frames after the last message.
         *
         * @default 15000
         */
        interval: 15000,
      },
    }),
  ],
})

Compression

A batch response might be left uncompressed by the Response Compression Plugin, depending on the shape it takes. On Node.js, add the Batch Response Compression Plugin when you know your batch responses are all compressible. It covers every one of them, without giving up streaming:

import { BatchResponseCompressionHandlerPlugin } from '@orpc/node'

const handler = new RPCHandler(router, {
  plugins: [new BatchHandlerPlugin(), new BatchResponseCompressionHandlerPlugin()],
})

Groups

Only requests in the same group are batched together. Each group also defines a context, as described in client context.

The following example batches requests by cache policy:

interface ClientContext {
  cache?: RequestCache
}

const link = new RPCLink<ClientContext>({
  method: ({ context }) => {
    if (context?.cache) {
      return 'GET'
    }

    return 'POST'
  },
  plugins: [
    new BatchLinkPlugin({
      groups: [
        {
          condition: ({ context }) => context?.cache === 'force-cache',
          context: {
            // used for the rest of the request lifecycle
            cache: 'force-cache',
          },
        },
        {
          // Fallback for all other requests. Keep this last.
          condition: () => true,
          context: {},
        },
      ],
    }),
  ],
  fetch: (url, init, { context }) =>
    globalThis.fetch(url, {
      ...init,
      cache: context?.cache,
    }),
})

Now, calls made with cache = 'force-cache' use that cache setting whether they are batched or sent individually.

Filtering Requests

Use filter to skip batching for specific requests before group matching runs. Requests for which filter returns false continue through the link chain individually.

const link = new RPCLink({
  url: '/rpc',
  plugins: [
    new BatchLinkPlugin({
      filter: ({ path }) => !path.includes('upload'),
      groups: [
        {
          condition: () => true,
          context: {},
        },
      ],
    }),
  ],
})

Wait Time

By default, only requests made in the same event loop tick are batched together. Set wait to keep collecting requests for a number of milliseconds, trading a little latency for fewer, larger batches.

const link = new RPCLink({
  url: '/rpc',
  plugins: [
    new BatchLinkPlugin({
      wait: 10, // ms
      groups: [
        {
          condition: () => true,
          context: {},
        },
      ],
    }),
  ],
})

The wait starts when the first request is queued, so no request waits longer than wait, even while more keep arriving.

Scope

On the server, requests from different users can be batched together. If your transport reads per-request data (in fetch, origin, and so on), set scope to a value unique to the incoming request.

import { AsyncLocalStorage } from 'node:async_hooks'

const requestStorage = new AsyncLocalStorage<Request>() // set per incoming request

const link = new RPCLink({
  url: '/rpc',
  plugins: [
    new BatchLinkPlugin({
      scope: () => requestStorage.getStore(),
      groups: [
        {
          condition: () => true,
          context: {},
        },
      ],
    }),
  ],
})

Response Headers

In buffered mode, the batch response carries the cookies and Vary fields of its subresponses, and the headers they all share. In streaming mode, it carries only the cookies of the first subresponse to respond. Other headers stay on their subresponse. To change what stays on each subresponse, set the server plugin’s mapSubresponse option.

Learn More

See the BatchHandlerPlugin source code and the BatchLinkPlugin source code for implementation details.

GeezerVison