Supplier feed sync for an EU Shopify retailer

Overview

I built a custom Shopify app that imports a wholesale supplier's XML feeds into an EU Shopify retailer's store and then keeps stock and prices current every hour; operators manage markup brackets, bulk actions, alerts and settings in a React dashboard.

It is client work that I designed, built and deployed alone, and it is live.

Context and problem

The retailer sells the supplier's catalog through its Shopify store, priced in euros from the supplier's cost in another currency, so the store has to follow the feeds on stock and price.

The client's requirements were to import the products and then new ones, update stock and prices every hour, manage markup brackets and bulk deletes from a dashboard, and build collections from the feed; translated product text came later as a change request.

The day-to-day users appear to be the store's operators, who manage pricing, bulk actions and alerts, while shoppers simply see current stock and prices.

My role

I was the only developer. All 32 commits are mine, with most of the work in the sync engine, dashboard authentication, the React dashboard, the API, the Shopify client and the plans.

I also worked with the client directly: I turned their requirements into a feature plan and recorded their answers to my open questions in the repo.

Constraints and requirements

• Shared, Basic-plan store: the store is on Shopify's Basic plan, so the sync paces itself on the GraphQL throttle status; other vendors sell through the same store, so the sync works at a single inventory location and never touches their products.
• Exact money: every price is calculated with decimal arithmetic, never floating point.
• Written rules: one sync at a time, diff before write, no automatic deletes and a fixed pricing order are set down as non-negotiable in the project rules and the README.
• Done means tested: a feature counts as done only when all its acceptance criteria are ticked and tests and lint pass.

Research and planning

Before the pricing and import work I listed open questions for the client, covering store currency and VAT, the feed schedule, zero-stock products, the compare-at price, the Shopify plan, inventory locations, variants with no stock row, how often to fetch exchange rates, and the app's auth mode, and I recorded each answer in the plan.

For variants missing from the stock feed, I tried several ways of matching them against the sample feeds before settling on treating them as out of stock.

The build then followed a feature-by-feature plan, F0 to F16, with acceptance criteria and a progress log; every planned feature is ticked.

Solution and user experience

Each run downloads the supplier's feeds, sanity-checks them and compares them with the stored catalog; only the differences are written to Shopify, by one job at a time.

The first import creates the catalog in bulk, an hourly run updates stock and prices, and a nightly reconciliation checks the store against the database.

Operators sign in to the dashboard with 2FA to set markup brackets, approve held exchange rates, run bulk actions, retry failed items, and read alerts and the audit log.

Key features

• Import: bulk productSet import, with products keyed by the feed's model ID metafield and variants by EAN; only changed items are sent.
• Hourly sync: the stock pushed is the supplier quantity minus a buffer, never below zero, and overselling is denied.
• Pricing: the cost is converted to euros with central-bank rates and two fallbacks, then VAT, a markup-bracket multiplier and a fixed divisor are applied and the price is rounded to end in 99 cents; a large exchange-rate jump waits for manual approval.
• Safety: feed sanity checks, a single-writer lock and no automatic deletes; a missing product goes to stock 0, then to draft.
• Reconciliation: a nightly comparison of a Shopify bulk export with the database.
• Collections: smart collections per category.
• Dashboard: 2FA login with a remembered device for 30 days, bulk delete, archive, block and restore, exchange rates, sync runs, error retry, email and Slack alerts, and an audit log.
• Translation: added as a change request; each unique feed text is translated once with OpenAI and cached, and a translations page in the dashboard manages them.
• Observability: the service exposes Prometheus metrics.

Architecture and stack

One Go binary serves the REST API, the embedded React dashboard and the background workers; a separate CLI runs migrations, user admin and manual syncs.

• Backend: Go with chi, pgx and sqlc, goose migrations, and River jobs on Postgres instead of Redis.
• Data: PostgreSQL 16 holds the catalog, pricing, runs and settings, and also backs the job queue.
• Shopify: the GraphQL Admin API only, at a pinned version, using productSet, bulk operations and staged image uploads.
• Installation: the app uses custom distribution in the client's organization with the authorization code grant.
• Dashboard: React, TypeScript and Vite with TanStack Query and Table and shadcn/ui, embedded in the Go binary.
• Testing and CI: GitHub Actions runs the Go tests and lint, Vitest for the dashboard and a Playwright end-to-end setup against a Postgres 16 service.
• Deployment: a docker compose stack on a VPS with Postgres, a migrate step, one distroless app image, Caddy for HTTPS and a backup sidecar, plus backup and restore test scripts.

Implementation details

The diff engine turns each run into a change set: creates, content, price and stock updates, missing and reappeared products, and blocked products to skip. Confirmed hashes are written only after Shopify accepts a change, so a second run on identical feeds produces no changes at all.

A failed item stays dirty and is retried on the next run; after three failed runs in a row, or at once on a permanent error, it is marked failed and waits for a manual retry from the dashboard.

Only one job writes to Shopify at a time, enforced with a Postgres advisory lock.

Dashboard passwords are hashed with argon2id and checked against a common-password list; 2FA uses TOTP with hashed single-use backup codes, sessions are stored as hashed tokens with a CSRF token, and repeated failures trigger a 15-minute lockout. Secrets kept in the database, such as the Shopify token and TOTP secrets, are encrypted with AES-256-GCM.

The runbook covers first deployment, upgrades and rollback, credential and key rotation, Shopify API upgrades, backups and restore, and what to do for each alert; a test fails if an alert kind has no runbook section.

Commit messages record versioned production deploys, from v1.2.0 to v1.3.1, within ten days of the first commit.

Challenges

• Full runs against a dev store surfaced stalled bulk operations, transient image failures, a metafield type issue and temporary token errors; I fixed each with cancel or retry logic.
• Some products were missing images until I moved from letting Shopify fetch them to uploading them directly, for new products too.
• The first deployment needed safeguards: a switch that turns background jobs off, and webhooks that follow the configured base URL.

Trade-offs

Running jobs on River inside Postgres likely keeps operations to one datastore and job state transactional with the data, at the cost of putting queue load on the main database.

Links

Client work: links and code are not public.

Skills demonstrated

• Shopify integration with the GraphQL Admin API, bulk operations and staged uploads.
• Backend development in Go, with PostgreSQL and a Postgres-backed job queue.
• Data pipeline reliability: diffing, locking, retries and reconciliation.
• Financial calculations with decimal arithmetic and exchange-rate safeguards.
• React and TypeScript dashboard development with secure authentication.
• Testing, CI, deployment and operations.
• Requirements gathering and client communication.
• An LLM translation step with per-text caching.