Work / Data pipeline · 2026
Shopify Uploader (catalog sync)
I designed and built this catalog pipeline alone for zweitliebe.de. Python scripts load the book catalog from supplier CSV feeds into Shopify through the GraphQL Admin API, track stock, price and compare-at price in SQLite change queues, and push batched updates, with stock running often and prices weekly. German genre collections are assigned by an OpenAI model whose answers are checked against the store's fixed collection list. The pipeline is live.
- live
- client work
- data pipeline
- Type
- Data pipeline · E-commerce integration: supplier feed → Shopify catalog sync with LLM classification
- Role
- Sole developer: designed, built and maintained the pipeline for the client.
- Built with
- Python, requests, python-dotenv, SQLite, Shopify GraphQL Admin API, OpenAI API, Bash, cron
- Last active
- 2026
What it does
- Initial product import with images and metafields, tracked by Shopify ID and content hash.
- Change queues for stock, price and UVP, applied in batches.
- Combined price and compare-at updates with tax fallback, .99 rounding and a zero-price guard.
- LLM genre-collection assignment validated against the store's collection map.
- Feed data clean-up: condition mapping, encoding, title length, author tags, negative stock.
- Frequent stock, weekly price and product import pipelines on cron.
How it is built
- Supplier feeds are downloaded into an SQLite state table that holds last and pending quantity, price and UVP with status flags, plus a log table. Separate mark and apply scripts per concern push batched GraphQL updates to Shopify, chained by bash pipelines on cron so stock runs often and prices weekly. An LLM step classifies new titles into the store's collections.
My part
- Built the Shopify GraphQL import and update scripts with throttle handling.
- Designed the SQLite change-queue model and the cron pipelines.
- Built and tuned the LLM collection classifier (prompt rules, exemptions, output validation).
- Fixed feed data issues over several months (conditions, encoding, titles, tags, prices).
Decisions
- Separate change queues per concern so stock and price run on different schedules.
- Constrain LLM output to known collection IDs, with NO_MATCH allowed.
- Guard cron overlap with flock (later removed from the products pipeline).
Challenges
- Messy supplier data: encodings, over-long titles, author tag commas, negative quantities and zero prices.
- Shopify API throttling during large updates.
- Getting reliable genre assignments from an LLM (novel logic, language and category exemptions).
Skills
- Shopify GraphQL Admin API (bulk updates, inventory, media)
- Scheduled data pipelines (cron, bash, SQLite state)
- LLM classification with constrained output
- Supplier feed data cleaning