Operations

Streams are long-lived gRPC calls; production consumers must assume they will drop and own their recovery. Four things cover almost everything: reconnect with backoff, handle DATA_LOSS, raise the receive size limit, and watch your block-height continuity.

Reconnect with backoff

The pattern: track the last block_height you processed, reconnect on any stream error with exponential backoff plus jitter, and reset the backoff once messages flow again. Never retry UNAUTHENTICATED; that is a key problem, not a transport problem.

import randomimport timeimport grpc def run_stream(subscribe):    """subscribe(last_height) opens the stream; replay from last_height    backfills the gap after a drop (Pro and above)."""    backoff = 1.0    last_height = None    while True:        try:            for msg in subscribe(last_height):                last_height = msg.block_height                handle(msg)                backoff = 1.0  # a healthy stream resets the clock        except grpc.RpcError as err:            if err.code() == grpc.StatusCode.DATA_LOSS:                # Your consumer fell behind the feed. Reconnect;                # subscribe(last_height) replays the gap.                pass            elif err.code() == grpc.StatusCode.UNAUTHENTICATED:                raise  # bad key, retrying won't help            time.sleep(backoff + random.uniform(0, backoff / 2))            backoff = min(backoff * 2, 30.0)

DATA_LOSS and replay recovery

A DATA_LOSS status means your subscription fell behind the feed; the server will not buffer unboundedly for a slow consumer. It is not a server fault and not a rate limit: the fix is on the consuming side.

  • Reconnect. The backoff loop above already does this; you resume at the live edge.
  • Backfill the gap (Pro and above). Resubscribe with replay from your last processed block height; the stream replays the missed range, then continues live with no gap. See Replay & backfill for the request shape per stream.
  • Fix the cause. Recurring DATA_LOSS means your handler is too slow for the subscription: narrow the server-side filters, move work off the receive loop, or split coins across consumers.

Max message size

Full order-book snapshots (the first message on StreamL4Book, and large raw blocks) can exceed gRPC's default receive limit. Set the maximum receive message size to at least 256 MB on any channel that subscribes to book snapshots or blocks. Otherwise the stream dies at the first snapshot with a message-too-large error.

channel = grpc.secure_channel(    "stream.hyperliquidrpc.com:443",    grpc.ssl_channel_credentials(),    options=[        # Full L4 book snapshots can exceed default limits (often 4 MB).        ("grpc.max_receive_message_length", 256 * 1024 * 1024),    ],)

Monitoring

Messages arrive in block order on every stream, which makes correctness cheap to verify:

  • Block-height continuity. Alert when block_height goes backwards or jumps unexpectedly. It is the one signal that catches every gap.
  • Staleness. Alert when a subscription that should tick every block goes silent beyond a few block intervals; check the stream demo to verify the feed shape before assuming it is on your side.
  • Reconnect frequency. Occasional reconnects are normal; a rising rate means an undersized consumer or network trouble.
  • Consumption. Watch per-key egress in the console against your tier's daily allowance. Overage semantics are on rate limits & tiers.
  • Keys. Rotate on a schedule, not just on incident. The zero-downtime sequence is in Authentication.