Loading
>
The skill file stays in the publisher's repository.
Paste the copied command in a terminal. It installs this SKILL.md into that agent.
npx skills add metalbear-co/mirrord --skill mirrord-header-propagation -a cursor -y
>
baggage header carrying mirrord-session=<key>)
end to end, so mirrord HTTP filters, preview environments, and queue splitting can route a
whole request chain to one developer's session. Detects existing OpenTelemetry, Datadog, or
other tracing that can already propagate baggage and enables it; otherwise adds a small
propagation layer. Covers inbound/outbound HTTP and gRPC, publishing to and consuming from
Kafka, RabbitMQ, SQS/SNS, GCP Pub/Sub, Azure Service Bus, NATS, Redis Pub/Sub, BullMQ and
Temporal, per-message context restore in consumers, and storage hand-offs such as a GCS signed
URL upload that triggers Pub/Sub or an S3 presigned upload that triggers SQS. Ends with a
report of every flow found and which ones still drop the header. Use when the user asks to
propagate headers, baggage, tracestate, the mirrord session key, or trace context across
services or queues, or when a mirrord filter matches the first service but not downstream
services or queue consumers.
metadata:
author: MetalBear
version: "1.1"mirrord routes traffic to a developer's session by matching a header. The convention is W3C
baggagewith amirrord-session=<key>entry: HTTP filters use"header_filter": "^baggage: .*mirrord-session={{ key }}.*$", and queue splitting filters on a message header or attribute namedbaggage. That only works past the first hop if every service forwardsbaggageon every outgoing call and every message it publishes, and every consumer restores it from each message it handles. This skill makes a codebase do that, preferring the observability stack it already has.
IMPORTANT: Follow these rules for all operations in this skill.
main, push, or open PRs unless the user asks.kubectl apply/patch, helm upgrade, terraform apply, or cloud CLI commands that modify buckets, topics, queues, subscriptions, or notifications. If a fix needs infra (e.g. a GCS notification payload format, SNS raw delivery), describe it for the user to apply.kubectl get, aws ... describe/get/list, gcloud ... describe/list, and reading env vars from manifests are fine. Don't read Secret values.baggage only for internal destinations (an allowlist of internal host suffixes such as .svc.cluster.local or the company's internal domains), and strip it on clients that call external hosts. Where the stack auto-injects into every client (e.g. a Java agent), strip baggage on the external clients or ask the user to strip it at the egress proxy. If neither is possible, don't ship that propagation without the user's explicit OK, and list it in the report.A flow is covered when a request carrying baggage: mirrord-session=alice at the edge results in mirrord-session=alice being present, under the name the mirrord filter reads, at every hop:
| Hop | Carrier | Name mirrord filters on |
|---|---|---|
| HTTP in/out | request header | baggage (http_filter.header_filter) |
| gRPC in/out | metadata | baggage (HTTP/2 header, same filter) |
| Kafka | record header | baggage (message_filter) |
| RabbitMQ | message headers table | baggage (message_filter) |
| SQS | message attribute (String) | baggage (message_filter, or jq_filter on .MessageAttributes) |
| SNS → SQS | SNS message attribute | needs sns: "true" on the queue, or raw message delivery |
| GCP Pub/Sub | message attribute | baggage (message_filter) — check the actual attribute name, see below |
| Azure Service Bus | application property | baggage (message_filter) |
| Redis Pub/Sub, BullMQ | no metadata — top-level JSON field in the payload | baggage (top-level field) |
| Temporal | activity/workflow header | header.baggage (message_filter) |
| S3 upload → SQS | object user metadata x-amz-meta-baggage | jq_filter on .S3Metadata.baggage with s3_event: "true" |
| GCS upload → Pub/Sub | object custom metadata baggage | jq_filter on .gcsMetadata.baggage with gcs_event: "true" |
Consumers must also put the message's baggage into the context of that message's processing, so outgoing calls and publishes made while handling it carry it on.
The operator's injectSessionKeyHeader stamps a mirrord-key header on traffic and messages it routes to a session. It is a marker of where a message was routed, not propagation: the app still has to forward baggage for the next hop to match. Don't treat it as a substitute.
Do the steps in order. Keep a running flow inventory (Step 1) — it becomes the final report.
Find every service in scope (monorepo dirs, Dockerfiles, Helm charts / k8s manifests) and its language. For each service, list:
ctx, asyncio.create_task / Celery tasks, in-memory queues, batch buffers that publish later.Find producers and consumers by searching for broker client usage (e.g. KafkaProducer, kafka.NewWriter, sarama, franz-go, confluent_kafka, kafkajs, @KafkaListener, amqp, pika, aio_pika, amqplib, RabbitTemplate, SendMessage, SendMessageBatch, ReceiveMessage, SqsListener, pubsub.NewClient, PublisherClient, SubscriberClient, @google-cloud/pubsub, ServiceBusClient, nats.Connect, bullmq, PUBLISH/SUBSCRIBE, generate_presigned_url, PresignClient, getSignedUrl, SignedURL, generate_signed_url). Cross-check destination names against env vars in the manifests and any MirrordSplitConfig / MirrordKafkaTopicsConsumer in the cluster (kubectl get mirrordsplitconfigs,mirrordkafkatopicsconsumers -A) so the inventory includes the queues people actually split.
Record each flow as source → carrier → destination (e.g. orders-api → Kafka orders.created → billing-worker).
Load references/libraries.md and check each service for an observability stack that can propagate W3C baggage. Signals: dependency manifests (go.mod, package.json, requirements*.txt / pyproject.toml, pom.xml / build.gradle*, Gemfile, *.csproj), startup wrappers (-javaagent:, opentelemetry-instrument, ddtrace-run, node -r dd-trace/init, NODE_OPTIONS=--require @opentelemetry/auto-instrumentations-node/register), operator injection annotations (instrumentation.opentelemetry.io/inject-*, admission.datadoghq.com/enabled), and env vars (OTEL_PROPAGATORS, OTEL_SERVICE_NAME, DD_TRACE_PROPAGATION_STYLE*, DD_TRACE_*_ENABLED).
For each service, decide which hops the stack actually covers. Three things commonly look covered but aren't:
baggage in OTEL_PROPAGATORS (or the programmatic composite propagator); Datadog needs baggage in its propagation style on the versions that support it; Micrometer Tracing only forwards baggage keys listed as remote fields.googclient_-prefixed attributes). Then either filter on that name or add an explicit baggage attribute.A service mesh (Istio, Linkerd) doesn't propagate headers for the app — the sidecar can't link an inbound request to the app's outbound call. The app must still forward baggage.
If every flow in the inventory is already covered, stop here: verify with Step 7 and give the "already covered" report (see Final report). Don't add code for its own sake.
Where a stack exists but a hop isn't covered, fix it in that stack before writing any custom code — per references/libraries.md:
baggage to the propagators / propagation style.mirrord-session to management.tracing.baggage.remote-fields and enable Kafka/RabbitMQ observation on templates and listeners.Where the env var lives in a Helm chart or manifest, edit it in the repo and tell the user it takes effect on the next deploy.
When a service has no stack, or its stack can't cover a hop, add the smallest layer that does. Load references/propagation-layer.md for per-language code. Rules:
baggage to that configuration (OTEL_PROPAGATORS, or the existing SetTextMapPropagator / set_global_textmap / setGlobalPropagator call). Installing a new global propagator would silently break the existing trace context.baggage only for allowlisted internal hosts.baggage value verbatim and merge rather than overwrite if the service adds its own entries. Never strip other members — other teams' tooling uses them.SendMessageBatch, Kafka batched sends, Pub/Sub batching) and retries/DLQ re-publishes. For SQS, add only the baggage attribute and check the projected count against the 10-message-attribute limit before changing the request. SQS rejects an over-limit send. If there's no room, send the message unchanged, log it, and record the flow in the report rather than dropping another attribute.baggage as a top-level field in the JSON payload — mirrord filters on top-level fields there. Make consumers tolerate the extra field.baggage (the stock OTel interceptor stores context under its own header key, which mirrord's header.<name> filter won't match).Every consumer must extract baggage from each message and run that message's handler inside a context carrying it, so its outgoing HTTP calls and publishes forward it. Check specifically:
@KafkaListener, @RabbitListener, @SqsListener, Celery, Sidekiq, NestJS microservices): confirm the framework's observation/instrumentation restores context, or wrap the handler.Bucket notifications are generated by the cloud provider, so they carry no request headers. Carry the baggage on the object and restore it from there. Load references/storage-handoffs.md; in short:
x-amz-meta-baggage into it from its current context, and the uploader sends that header. S3 event notifications don't include user metadata, so the consumer calls HeadObject and restores baggage from it. For splitting, set s3_event: "true" (plus sns: "true" if via SNS) on the queue's queueConfig and filter with jq_filter: ".S3Metadata.baggage // \"\" | test(\"mirrord-session=alice\")".x-goog-meta-baggage into the URL and the uploader sends it. For splitting, set gcs_event: "true" on the queue's queueConfig (operator 3.214.0+) and filter with jq_filter: ".gcsMetadata.baggage // \"\" | test(\"mirrord-session=alice\")" — this works with either notification payload format, since with NONE the operator fetches the object's metadata from Cloud Storage itself (needs storage.objects.get on the bucket for the client config's identity). For a consumer that restores context in-process rather than relying on mirrord's filter, the path differs by payload format: with JSON_API_V1 the message data is the object resource including metadata, so restore baggage from .metadata.baggage; with NONE the consumer has to fetch the object's metadata itself, the same as the operator does.baggage header.Use the narrowest check that proves each changed flow:
baggage, and a consumed message with baggage produces outgoing calls that carry it.kcat -C -t <topic> -f '%h %s\n' -o -5 -e; SQS aws sqs receive-message --message-attribute-names All --visibility-timeout 0 (warn that receiving still counts toward the redrive policy); Pub/Sub — inspect via a test subscription the user creates, don't create one yourself; RabbitMQ management UI "Get messages" with requeue.mirrord-session=<key> (HTTP header_filter or queue message_filter) and send curl -H "baggage: mirrord-session=<key>" at the edge. Point to the mirrord-config, mirrord-kafka, and mirrord-prev-env skills for that config.If you can't run any verification, say which flows are unverified.
Always end with this report. Exactly one of two shapes:
Everything already covered:
✅ baggage already propagates across all <N> flows found — no changes needed.
Stack: <e.g. OTel Java agent 2.x with tracecontext,baggage; kafka-clients and AWS SDK instrumented>
Verified: <how, or "not verified: <why>">
Otherwise:
Flows: <covered before> already covered · <changed> fixed · <remaining> not covered
| Flow | Carrier | Status | How |
|------|---------|--------|-----|
| web → orders-api | HTTP | ✅ already covered | OTel Node auto-instrumentation |
| orders-api → orders.created → billing-worker | Kafka header | 🔧 fixed | added @opentelemetry/instrumentation-kafkajs |
| uploads-api → S3 → uploads-queue → thumbnailer | S3 metadata | 🔧 fixed | signed x-amz-meta-baggage; HeadObject in consumer |
| nightly-reindex (cron) → search-indexer | Pub/Sub | ❌ not covered | no inbound request to carry a session |
Not covered:
- <flow>: <why> — <what would fix it, and who has to do it (app team / infra / third party)>
The Not covered list is mandatory whenever anything remains. Typical entries: cron/scheduled jobs and other flows with no inbound request; third-party webhooks and direct third-party uploads; SQS messages already at 10 attributes; GCS notifications with NONE payload; brokers or clients with no instrumentation where the user declined a code change; external egress where baggage is deliberately not forwarded; in-process hand-offs you found but couldn't safely change. Also list anything inferred but not verified.
baggage header with only mirrord-session — forward and merge.baggage to it.baggage into calls to external hosts.message_filter key that reads it.mirrord-key to propagate for the app.