How Data Purl validates its data
Data Purl checks its data at every step. Over the last two years, 95.9% of scheduled retailer-weeks were captured without a gap, and every gap is filled from the last good week and flagged. Every product passes outlier screens before it is published, and 91% of Hardlines & Consumables products have a high-confidence category.
Figures as of 26 September 2026, read from the production pipeline and the classification audits.
See a worked example: what we capture, three retailer pages beside their Data Purl records, field by field.
1. Collection
Retailer and brand websites are collected several times a week and published weekly.
- Captured without a gap: 95.9% of retailer-weeks.
- Gap-filled: 4.1%. A week counts as a gap when a retailer returns no data or fewer than a quarter of its usual number of SKUs. The gap is filled with that retailer's last good week and flagged (IS_FILLED), so it can be excluded from any analysis.
- Capture frequency: each product is captured on about 4 days a week on average; 54% of retailer-weeks are captured on five or more days and 26% every day.
- Data load jobs retry automatically and alert the operations team when they fail.
By market
| Market | Feeds | Gap-free | Days a week | Five or more days |
|---|---|---|---|---|
| United States | 137 | 94.9% | 4.0 | 53% |
| United Kingdom | 134 | 97.6% | 4.6 | 66% |
| Germany | 38 | 94.9% | 3.3 | 40% |
| Spain | 27 | 97.3% | 3.8 | 51% |
| Australia | 10 | 97.5% | 2.1 | 22% |
| South Korea (since April 2025) | 5 | 82.2% | 1.0 | 0% |
| India (since April 2025) | 5 | 82.1% | 1.0 | 0% |
| Japan (since March 2025) | 4 | 88.3% | 1.0 | 0% |
| Hong Kong (since June 2025) | 2 | 87.8% | 1.0 | 0% |
| South Africa | 1 | 100% | 1.0 | 0% |
Newer Asia-Pacific feeds (started in 2025, month shown) are collected once a week, so a missed run is a missed week. That is why their gap-free share is lower.
By sector
| Sector | Feeds | Gap-free | Days a week | Five or more days |
|---|---|---|---|---|
| Apparel, footwear and luxury | 304 | 95.6% | 3.6 | 47% |
| Hardlines & Consumables | 61 | 97.2% | 5.9 | 87% |
Last 104 weeks. Gap-free: share of scheduled retailer-weeks captured without a gap. Days a week: average days each product is captured in gap-free weeks. Five or more days: share of gap-free retailer-weeks captured on at least five days.
2. Quality screens before publication
Every product-week passes automated screens before it reaches clients. A product-week is held back when:
- It was seen on fewer than four days in the week.
- Its discount is above 80%, or its selling price is above its full price.
- Its price is an outlier for its category and market (more than three interquartile ranges from typical prices, on a log scale, where the category has at least 20 products).
- It has no category or no price.
Thin retailer and category combinations are also held back, so a handful of listings cannot move a category figure. Automated tests check that product keys are unique, that weekly records are not duplicated, and that brand ownership dates never overlap.
3. Classification
Every product is placed in one standardized tree of about 4,000 categories. 91% of Hardlines & Consumables products in scope have a high-confidence category: one that has been individually reviewed and confirmed, or assigned by audited classification rules. The rest are classified by a model, used only where its calibrated confidence is at least 0.75. Fashion products are classified by an analyst-maintained rule model.
Brands are resolved to their owners, public or private. Listed owners are mapped to tickers with dated start and end dates, including licensing arrangements, so each observation is attributed to the owner at the time it was made.
4. Freshness
- Weeks run Monday to Sunday.
- The weekly production job starts at 1pm New York time on Wednesday and completes by 4pm, delivering the Monday-to-Sunday week that has just ended.
- A price change or sell-out on a retailer's site is usually captured within a few days, and reaches clients between about three days and two weeks after it happens, depending on when in the week it happens.
- When history is restated, the publish timestamp on each row changes, so clients can see which records moved.
5. Backtests
Data Purl tests whether its shelf data tracks the numbers companies report. For each covered company, a model turns year-on-year changes in Data Purl measures (average listed price, markdown breadth, active SKUs and core mix) into a signal for the company's reported gross margin, and each model is re-scored every quarter against the reported result. Four examples:
Brands are attributed to the company that owned them at the time of each observation, so acquisitions and disposals do not leak future information into the history. Models for other companies, and current-quarter signals, are available to clients. Not investment advice.
What the data does not measure
Data Purl observes what retailers list online: prices, discounts, availability and assortment. It does not observe transactions, unit sales, checkout-only or loyalty discounts, or physical stores. See the methodology for full definitions.
The evidence on this page (history collected the same way every week, flagged gaps, audited classification and backtests) is what a one-off AI scrape cannot produce. See Data Purl vs AI web scraping.
Need more detail for due diligence?
We answer vendor due diligence questionnaires and can walk your team through collection, classification and quality controls.