Replacing a Rented Marketplace Data Supply Chain
An Azure landing zone and modeled warehouse built first, with existing reporting repointed at it, then three rented acquisition routes rebuilt as ingestion the agency owns: first-party API connectors, a direct retrieval service, and scheduled browser automation.
- Data Engineering
- Data Platform
- Marketplace Ingestion
- Microsoft Azure
- Three
- Rented acquisition routes replacedA subscription connector service, a per-call product data provider and a desktop automation tool, each replaced by ingestion the agency owns and can extend. A count of dependencies removed, taken from the delivery record. It is not a saving: no cost, cycle time or runtime figure was measured on this engagement, and none is published here.
- Seven
- Seller data categories, first-party connectorsSales, inventory, product information, advertising metrics, forecasts, market insights and purchases, each built against the marketplace's own seller API on Azure Functions. A scope count from the delivery record rather than a performance measure.
Overview
For this agency the reporting is the product. Brand strategy, audience targeting, content optimization, paid search management and social campaign work are all arguments made from marketplace data, so a late number is a late client deliverable and a contested number in a client review is a credibility problem.
Almost none of that data arrived under the agency's own control. Three acquisition routes were rented, and underneath them there was no authoritative place for anything to land: feeds terminated in four separate storage platforms and were reconciled by hand.
The first engagement built no connectors at all. It built the landing zone, the modeled warehouse and the reconciliation logic, then repointed the existing reports at the modeled layer. Only once that boundary existed were the rented routes rebuilt, each matched to what its source actually exposed.
The Context
The client is not named. The engagement was delivered for a US performance marketing agency selling brand strategy, audience targeting, content optimization, paid search management and social campaign work to a portfolio of Amazon marketplace sellers.
Its evidence base is marketplace data: sales, inventory, product attributes, advertising performance, demand forecasts and category-level competitive signals. Data acquisition was not a back-office function there. It sat directly underneath the product.
The estate had grown the way these usually grow. A connector service was bought because buying was faster than building. A product data provider was subscribed to because it answered a question nothing else could. An automation script was written because a category browse existed only in the interface. Each decision was locally reasonable. Together they produced a supply chain the agency neither owned nor could reshape.
The Challenge
Three rented routes, each failing differently, and a fourth problem underneath that made all three permanent.
- A connector subscription that could not be shaped to the agency's schema and shipped duplicate records it then had to clean
- A product and keyword data provider charging per call, with identifier and search term lists assembled and updated by hand
- A desktop automation tool walking browser node identifiers held across spreadsheets, several passes per product, with reporting waiting on it
- Feeds terminating in Google BigQuery, Amazon S3, OneDrive and Google Cloud Storage, reconciled by hand, producing numbers that did not always agree
The per-call contract carried the cost nobody had budgeted for. Because every lookup was billable, analysts limited what they asked. Curiosity was metered, and that never appeared on an invoice.
What Was at Stake
When a vendor owns your ingestion, a price change, an API change or a service withdrawal is your outage, and there is no fallback that is not a project. For an agency whose deliverable is the report, that lands on client commitments rather than on an internal system.
Next to it sits an assurance problem. Numbers reconciled by hand across four storage platforms cannot be defended line by line in a client review, and where the reporting is the product, a contested figure is a commercial event rather than a data quality ticket.
There is a custodial obligation too. An agency holds credentials for seller accounts it does not own, and handling those informally is the kind of exposure that ends a platform relationship as well as a client one.
The last piece is the trap in the obvious plan. Replacing the worst feed first produces a well-built connector writing into the same fragmented estate, because there is still nowhere better for it to land.
Our Role
Kainskep delivered the platform and the acquisition tier across four engagements run as one program. The order was the design decision: the landing zone, the modeled warehouse and the reconciliation logic came first, and no connector work started until reporting had been repointed at the modeled layer.
- The Azure landing zone, the staged, transformed and curated layers, and the reconciliation and validation steps in staging
- Custom connectors against the marketplace seller API, running as Azure Functions in Node.js
- A Python retrieval service replacing a paid intermediary, with a React interface over it
- Browser automation rebuilt in Python and Selenium, scheduled and monitored on a server rather than a desktop
- Secret handling, role-based access and the administration portal over the ingestion tier
The agency kept what it should keep: its seller relationships and the credentials that come with them, its report definitions and the people who read them, and the judgment about which products and keywords matter. The node identifiers stayed in the spreadsheets that already held them, and the automation was built to read those rather than asking the business to change how it worked to suit the system.
What We Did, and in What Order
Four steps. The order carries more of the result than any individual choice of tooling in it.
Build the floor before touching a feed
The first engagement built no connectors. Azure Data Factory collected from the sources that already existed, raw and semi-structured output landed as Parquet in Azure Data Lake Storage Gen2, and a modeled layer in Databricks SQL, queried through Databricks notebooks, reproduced the transformation logic that had been living inside ad hoc data flows, with reconciliation and validation steps so that a figure could be traced rather than argued about.
Repoint the reports, then stop
Existing Power BI reports were moved onto the modeled layer. That step is small in a plan and large in effect. Once reports read one modeled source they have no relationship with whatever fetched the data, and every later change to acquisition is invisible to the people consuming the output.
Match the mechanism to what each source exposes
Only then were the rented routes rebuilt, and deliberately not to a single pattern. A documented first-party API, a paid intermediary and a rendered interface are three different problems. One ingestion pattern across all three either over-engineers the easy cases or fails the hard one.
Design for the source saying not yet
Marketplace reports are generated on the platform's schedule, not the caller's. Retrieval runs on dynamic triggers against defined intervals, with an Azure Service Bus queue holding trigger events and an availability check before a run is committed, so a report that is not ready is re-attempted rather than lost.
This was four engagements and one program. Doing the acquisition work first would have produced three well-built connectors writing into the same fragmented estate, which is the most common version of this mistake and the most expensive to undo.
Three Acquisition Patterns, One Landing Contract
The standard is where data lands, not how it is fetched. Everything upstream of the landing layer is replaceable, everything downstream of the modeled layer is stable, and that boundary is the deliverable.
Where a first-party API existed, connectors were built against it
Custom connectors across seven categories of seller data: sales, inventory, product information, advertising metrics, forecasts, market insights and purchases. Azure Functions in Node.js handle retrieval and processing, built to the marketplace's own contract rather than to a third party's interpretation of it, which is what removed the duplicate records the subscription service had been introducing.
- Azure Functions
- Node.js
- Seven data categories
Where only a paid intermediary stood in the way, it was removed
A Python retrieval service works directly from product identifiers and search terms, normalizing output before it reaches the lake, on Azure App Service with a React interface so the people who know which products matter maintain the lists themselves. The change is commercial more than technical: with per-call charging gone, the marginal cost of one more question falls close to nothing.
- Python
- Azure App Service
- React
Where data existed only in the interface, automation was engineered rather than recorded
The category browse driven by node identifiers has no API path, so this is scraping and is named as such. It was rebuilt in Python with Selenium on a scheduled job hosted on Azure App Service, handling authentication and challenge-response verification as part of the flow and reading node identifiers from the spreadsheets that already held them. Extracts land in the lake and are processed through Databricks into the warehouse layer.
- Selenium
- Scheduled and monitored
- Databricks
Absence treated as a normal condition
Dynamic triggers schedule retrieval against defined intervals and an Azure Service Bus queue holds trigger events, so a report the marketplace has not generated yet is re-attempted rather than dropped. Retrieval and storage activity is logged and monitored. A gap in the marketplace's readiness stops being a gap in the agency's reporting.
- Service Bus
- Availability checks
- Logged retrieval
Credentials for accounts the agency does not own
Seller account keys sit in Azure Key Vault and are retrieved by the functions at call time. A React administration portal over the ingestion tier carries authentication, role-based access control and user profile management, so administrators add and manage seller accounts, monitor retrieval and see system health without going near the code.
- Key Vault
- Role-based access
- Azure AD
One modeled layer, and one place a figure comes from
Staged, transformed and curated Parquet layers in the storage account, modeled and queried in Databricks SQL through Databricks notebooks, with reconciliation and validation in staging and Power BI reading the modeled layer alone. Pipeline success and failure raise alerts through Logic Apps, so a silent failure becomes a noticed one, and changes ship through Azure DevOps.
- Databricks SQL
- Power BI
- Logic Apps
Browser automation is usually the wrong answer, and it is worth saying so before a reader does. It was the right answer on one surface here because the alternative was not a cleaner API, it was no data. What makes it defensible is that it runs as scheduled, monitored, server-side infrastructure with a defined landing point rather than as a macro on somebody's desktop. If that surface ever gets an API, a connector replaces it and no report changes.
What We Delivered
The landing and modeling platform
Azure Data Factory collection from the sources that already existed, an Azure Data Lake Storage Gen2 landing layer holding Parquet, and staged, transformed and curated layers in Databricks SQL, queried through Databricks notebooks, carrying the reconciliation and validation logic.
Marketplace API connectors
Seven categories of seller data retrieved and processed by Azure Functions in Node.js, with dynamic triggers and queued trigger events behind them.
A direct retrieval service
Python on Azure App Service working from product identifiers and search terms, with a React interface for maintaining those lists without an engineer.
Engineered browser automation
Python and Selenium on a scheduled job hosted on Azure App Service, with authentication and challenge-response handling, landing extracts in the lake for Databricks to process.
An administration portal
Authentication, role-based access control and user profile management over the ingestion tier: seller accounts added and managed, retrieval monitored, system health visible to an administrator.
Secrets, alerting and a delivery pipeline
Seller account keys in Azure Key Vault, access governed through Azure AD, pipeline outcomes alerted through Logic Apps, observability through Azure Monitor and Security Center, and CI/CD through Azure DevOps.
Nothing here was migrated. The acquisition tier was rebuilt underneath a reporting layer that kept working throughout, and the reports themselves were repointed once and then left alone.
Outcomes
What changed is structural, and it is stated that way here because that is what the engagement record supports.
Acquisition moved from three external dependencies to systems the agency owns and can modify. Reporting sits behind one modeled source, so ingestion can be changed, extended or replaced without a report changing. Reconciliation moved out of manual effort into the pipeline, with validation in staging and alerting on pipeline outcomes. Marketplace unavailability stopped being a reporting gap, because trigger events queue and re-attempt. Seller credentials moved into a managed secret store behind role-based access. Adding a data category or a seller account became a configuration and operations task rather than a development one.
No percentage, cost saving or runtime figure appears on this page, because none was measured. The connector and per-call spend that came off the bill, and the change in acquisition cycle time, are the numbers that would matter, and neither was instrumented before or after. The structural changes are demonstrable; a figure that falls apart the first time someone asks how it was calculated is worth less than the page it sits on.
- Three rented acquisition routes replaced by ingestion the agency owns
- Reporting decoupled from acquisition behind one modeled layer
- Reconciliation and validation in the pipeline, with alerting on outcomes
- Seller credentials in a managed secret store, behind role-based access
The test a reader can apply to their own estate is one question long: if you had to replace a data provider next month, how many reports would have to change? That number is the entire argument for doing the platform work first.
What This Page Does Not Cover
Published in anonymized form. The client, the displaced vendors and the tools they sold are not named, and neither is any other firm involved in delivery.
Deliberately not published, in the interest of being accurate rather than complete: any percentage improvement, cost saving or runtime figure, because none was measured on this engagement; the commercial terms of the subscription and per-call contracts that were replaced; seller account volumes and record counts; and the agency's schema, identifier lists and report definitions, which are its own.
Two wording points, because both are easy to get wrong in either direction. The browser automation described above is scraping, not API integration. And nothing was migrated: acquisition was rebuilt and reporting was repointed.
Related Capabilities
The disciplines this engagement drew on, and where each is described in full.
Data Engineering & Machine Learning
The landing zone, the modeled warehouse, and the reconciliation and validation that let a figure in a client report be traced rather than argued about.
Data engineering and machine learningCloud Infrastructure
The Azure estate underneath it: storage, App Service, queueing, secret handling and the alerting that turns a silent pipeline failure into a noticed one.
Cloud infrastructure engineeringApplication Development
The connectors, the retrieval service and the React administration portal, including the role-based access that makes adding a seller account an operational task.
Software engineering for complex applicationsDevOps & Automation
Acquisition that runs as scheduled, monitored infrastructure rather than as scripts on a desktop, shipped through a pipeline rather than by hand.
Delivery and operations engineering