EDI / Integration
2026

Portfolio

Building resilient data pipelines

EDI mapping, API ingestion and multi-system integration — engineered to survive high-volume, edge-case, failure-heavy production traffic without dropping a document or duplicating an order.

X12EDIFACTAS2PythonSFTPRESTOAuth2QueuesWebhooksIdempotencyBatchingMonitoringERPWMSTMS

Where throughput sticks

TRACE
BOTTLENECK: MAPtap a stage
Mapbottleneck

In

210/s

Out

40/s

Depth

14k

Per-partner transforms plus live item and ship-to lookups. This stage runs five times slower than the one feeding it, so everything behind it starves.

Cache code lookups, precompile templates, shard mappers by partner, then scale the worker pool — in that order, because the cache is the cheapest win.

ingest
3.1k
parse
1.9k
map
14k
queue
14k
transmit
6.8k

Ingest 220/s vs. map 40/s — the queue grows about 180 documents a minute until something upstream slows down.

Throughput — batch vs. individual

MODE
2,400documents / hour

Per doc

0.3s

API calls

4%

Cost index

13%

Documents grouped into one envelope per partner window. One handshake, one acknowledgement, one slot in the partner's rate limit — the right call when the SLA is in minutes.

7.7×more throughput when the partner's SLA allows batching.

Worked example — one 850, three systems

X12 → JSON

Raw 850 · positional segments, tilde delimiters

ISA*00* *00* *ZZ*ACMESUPPLY *ZZ*RETAILCORP *250926*1200*U*00401*000000001*0*P*>~
GS*PO*ACMESUPPLY*RETAILCORP*20250926*1200*1*X*004010~
ST*850*0001~
BEG*00*SA*PO-48213**20250926~
N1*ST*DC CHICAGO*92*DC01~
PO1*1*48*EA*12.75**VP*SKU-8841~
PO1*2*120*EA*3.40**VP*SKU-1042~
CTT*2~
SE*8*0001~
GE*1*1~
IEA*1*000000001~

This is what the partner sends — envelope, control numbers, positional segments. Useful to machines, unreadable to the ERP. The parser walks it element by element, including the escaped delimiters, and rejects anything malformed here.

Skill segments

L1·0080
Map

X12 & EDIFACT mapping

850, 855, 856, 810 and 860; ORDERS, ORDRSP, DESADV, INVOIC. Element and qualifier level, loops and control segments included — not a black-box translator.

Parse

Python data parsing

Envelope-aware tokenizers over flat, positional and delimited files. Escaped delimiters, embedded newlines and ragged loops become typed records.

Ingest

API ingestion

OAuth2 client-credentials, cursor pagination and rate-limit headers respected. Backfills run through the same path as live traffic.

Push

Webhooks

HMAC signature checks, timestamp windows and fan-out to consumers, so one partner event can drive several systems at once.

Guard

Idempotency

A dedup key per document means at-least-once delivery never turns into duplicate orders. Replays and retries are safe by construction.

Bridge

Multi-system integration

ERP, WMS, TMS and 3PL portals behind one canonical contract. One source of truth per document, no side spreadsheets.

Watch

Monitoring

Per-stage latency, queue depth, acknowledgement rate and dead-letter volume on one board, with alert thresholds per partner SLA.

Recover

Error handling

Validate first, quarantine second, and hand a human a reason code they can act on. Negative acknowledgements route back to the sender.

Monitoring — queue depth

LAST 1H
LIVE4 queues
asn.in
3,100
po.out
6,800
map.workers
14,000
dlq
212

map.workers above 10k for 12m — paged. Threshold is per partner SLA, not one global number.

Error handling — what happens when

POLICY

Syntax — missing segment, bad delimiter

Reject at parse, return a 997 with the failing element id.

Semantic — unknown SKU, PO not found

Quarantine with a reason code a buyer can actually action.

Transport — timeout, TLS, partner down

Retry with exponential backoff and jitter, capped, then alert.

Duplicate — same key, second arrival

Acknowledge and drop. The first write already won.

Poison — fails after N attempts

Move to the dead-letter queue, one click to replay after the fix.

0 duplicatesevery rule above is written so a retry cannot double-book an order.
AINext-step

Where models raise the ceiling

AI is a force multiplier on the slow, repetitive parts of integration work — the drafting, the triage, the test data. The judgement about what a trading partner's data means still sits with me.

01

Auto-draft mappings

Point a model at a new partner's 856 sample and it proposes the field map. I verify element by element before anything ships — it removes the blank page, not the review.

02

Failure triage

Cluster dead-letter dumps into root causes, rank them by volume and draft the fix, so the queue clears from the biggest reason down instead of the loudest complaint.

03

Edge-case generation

Synthesize malformed and awkward envelopes — split loops, stray qualifiers, midnight cutover dates — so the regression suite covers the corners production actually hits.

04

Plain-language runbooks

Ask why a partner's acknowledgements stopped at 03:00 and get the trace summary plus the next command to run, instead of digging through five log sources.

How I keep improving

LOOP

Measure the stage before touching it — a bottleneck fixed by guesswork comes back as an outage.

Treat every dead-letter item as a missing test case, so the same failure only happens once.

Keep one SLA board per partner; averages hide the account that is about to escalate.

Write mappings as reviewable code, so the next engineer can read the intent and not just the output.

ConnectMessage me on LinkedIn→or email you@example.com