WFL Group PIM to Shopify pipeline

Overview

For WFL Group I built the tooling between the client's product information system (PIM) and Shopify: it takes the per-country product exports, re-hosts the product images on a CDN storage zone per country, rewrites the image URLs in the export, and imports the products with their variants into Shopify.

It was freelance work, and the files date from September 2024 to May 2025.

Context and problem

The client's PIM is Akeneo with a CSV export connector, and it writes a separate export for each storefront country: Germany, Spain, France, Italy, the Netherlands and the UK.

The exports carry image URLs, and the pipeline moves every image to a CDN storage zone for its country before the import, which suggests each storefront needed its images on storage it controls.

The pipeline appears to serve WFL Group's per-country Shopify storefronts.

The PIM appears to be the client's own system, which I wrote around rather than in.

An older spider, dated November 2024, targets an earlier Ergonode-based setup; the six country spiders from May 2025 read the Akeneo exports.

My role

This was freelance work for WFL Group. I wrote the three parts around the PIM: the watcher and copy scripts on the PIM server, the per-country Scrapy spiders that re-host the images, and the Lumen importer that loads Shopify.

There is no git history, so the dates come from file times, and no other contributor is visible in the folder; I appear to be the only author.

Constraints and requirements

• Six storefront countries (DE, ES, FR, IT, NL and UK), each with its own export file and its own CDN storage zone.
• The goals: hand off each new export automatically, move the images to CDN storage without uploading them twice, and import the products with their variants into Shopify.

Solution and user experience

Judging by the files, the pipeline runs in four steps: the PIM writes an export, a watcher publishes the latest per-country CSVs, a spider per country re-hosts the images and rewrites its CSV, and the Lumen importer loads the result into Shopify.

The hand-off starts on its own whenever the PIM writes a new export, and each importer step is called through its own route on the Lumen app.

Key features

• Export hand-off: a watcher reruns a copy script whenever the PIM writes a new export, publishing the latest DE, ES, FR, IT, NL and UK export CSVs and a linker file to a public export folder.
• Image re-hosting: for each SKU, the spider downloads every image in the export's image columns, uploads it to that country's CDN storage zone and swaps the URL in the CSV; an uploaded.csv ledger skips files that are already uploaded.
• Category flattening: category paths are cut to their last segment while the rows are rewritten.
• Variants: PIM rows are grouped into parent products and variants, with Size and Color options built from the export's size and colour columns.
• Product copy: table specification fields, such as tabletop material, diameter and leg length, are added to the description as a Specifications block.
• Image queue: product images go through a queue table that is processed 1,000 rows at a time.
• Updates and resets: an update step refreshes price, cost and stock, and helpers can delete and reload the whole catalogue.
• Pricing: a price markup helper that I share with my other 2024 importers.

Architecture and stack

The work is split into three parts: three shell scripts for the PIM server, seven Scrapy projects (one per country plus one for the older PIM), and a Lumen importer.

• Server automation: Bash scripts with inotifywait watch the PIM's export folder.
• Image transfer: Python and Scrapy, with a custom spider that chains requests, reads the CSV through itertools.tee, uses the validators library and handles Scrapy's spider_closed signal.
• CDN storage: Bunny.net Storage over its HTTP API, with PUT uploads and DELETE.
• Importer: PHP and Lumen 9 with the Laravel HTTP client, calling the Shopify Admin REST API for products, variants, images, metafields and inventory.
• PIM: Akeneo, which appears to be the client's system rather than something I set up.

Implementation details

The watcher refuses to start a second copy of itself, then watches the export archive recursively and reruns the copy script on every change.

The copy script picks the newest copy of each of the seven target files by modification time and copies it to the public export folder, logging any file it cannot find.

The importer stages the export in a local table and tracks every row with a status: waiting for upload, synced, or waiting for an update, with separate codes for failed creates and duplicates. A second table maps each parent SKU to its Shopify product ID and the variants Shopify returned.

Images are not sent with the product. Once Shopify has created the product, each variant's image URLs are queued with the variant ID Shopify returned, and a separate step attaches each image to its variant.

The update step checks the stored product first: if Shopify no longer has it, the mapping is dropped and the row goes back into the upload queue; otherwise the price, cost and stock are updated, with a one-second pause between products.

Trade-offs

Each country has its own copy of the spider, with its own CSV name and storage zone, rather than one spider with a country parameter; that was likely quick to set up per country, but it duplicates the code across six projects.

Scrapy reads a CSV here, not web pages, so it seems to have been chosen as an asynchronous HTTP pipeline for moving images in bulk rather than as a scraper.

The hand-off is triggered by a file watcher instead of a fixed schedule, which likely means a new export moves on as soon as the PIM writes it.

The uploaded.csv ledger lets a rerun skip images that are already on the CDN instead of uploading them again.

Links

No public link or repository for this project.

Skills demonstrated

• PIM-to-commerce data pipelines across six country storefronts.
• Bulk media transfer to CDN storage with Python and Scrapy.
• Server automation with Bash and inotify.
• Shopify product import with variants, options, images and inventory through the Admin REST API.