Methodology: how Data Purl collects, standardizes and measures eCommerce data

Product listings are collected several times a week from 400+ retailer and brand websites across the US, Europe and Asia, and published weekly. How collection is checked is set out on the validation page. Only publicly available product information is collected: product names, prices, discounts, availability and product attributes. No personal data is collected.

Each observation records the product's listed price, its full (original) price where shown, and whether it is available to buy. Products carry first-seen and last-seen dates, which support newness and delisting analysis.

Retailers organize their sites differently, so Data Purl places every product into one category tree of about 4,000 categories, organized by department, category group and category down to a specific product type. Classification uses product names, retailer breadcrumbs and product attributes, and is audited for false positives before release.

When the tree or the classification rules improve, the full history is reclassified under the current rules. Series are therefore restated rather than frozen: a figure produced today may differ slightly from the same figure produced a year ago. Published figures state the date they were produced.

Brand names are normalized across retailers (spelling, punctuation and sub-brand variants) and resolved to a single master brand. Master brands are mapped to their owning company and, where the owner is listed, to its security identifiers. Licensing relationships are recorded separately from ownership.

Ownership and licensing changes are dated. Historical observations are attributed to the owner at the time of the observation, which avoids look-ahead bias in backtests.

Pack sizes are parsed from product names and attributes into a net size and a unit of measure (for example ounces, grams, liters or count). This allows prices to be compared per unit, which is required to capture shrinkflation: a smaller pack at the same shelf price is a price increase per unit.

What each term means:

• Product: one item as a retailer lists it, such as a style of shoe in one color.
• SKU: each size or variant of a product. Breadth and in-stock measures count SKUs.
• In stock: available to buy on the day it was observed.
• Full price: the retailer's own before-markdown reference price. A SKU is marked down when it sells below it.
• Week: Monday to Sunday. Weekly figures average the days observed.

Core metrics are computed from the standardized weekly observations:

• Markdown breadth: the share of each product's in-stock sizes and variants selling below full price, averaged across products so each product counts equally.
• Discount depth: average percentage reduction from full price across discounted SKUs.
• In-stock rate: share of listed SKUs available to buy.
• Assortment count: number of distinct active products, at style or SKU level.
• New-product share: share of active products first seen within a recent window.
• Matched-SKU inflation: price change of identical products observed in both periods.
• Mix shift: the part of an average-price change explained by changes in which products are on sale.

Averages are simple (unweighted) by default, so a single high-priced product cannot dominate a group. Where a weighting is applied it is stated. Full definitions are in the glossary. The free Shelf Indices publish three of these measures monthly by department, and full-price change for 30 well-known categories, each with its definition.

Data is weekly. Year-on-year comparisons use the same calendar weeks in the prior year to remove seasonality, and can be aligned to a company's fiscal calendar. Promotional events that move between years should be checked before reading a single week's change.

The full history is rebuilt every week. Data Purl does not append new weeks onto frozen old ones: each weekly build recomputes history back to 2013 from the underlying observations, so improvements to cleaning and classification apply to the whole history, not only to new data.

Categories are assigned per product, not per week. Each product carries one current category in the standardized tree, applied to every week it was observed. When a product is reclassified, or the tree itself changes, its whole history moves with it, so category series never break. The trade-off: past category figures can shift slightly after a reclassification.

Ownership links are point-in-time. The links from a brand to a listed owner, from a retailer to its ticker, and licensing deals all carry start and end dates, applied week by week. A brand counts toward an owner only for the weeks that owner held it, so acquisitions and divestments are dated and there is no look-ahead bias.

Gaps are flagged, not hidden. A missed retailer-week is filled from that retailer's last good week and flagged (IS_FILLED), so it can be excluded.

By dataset. Structured Datafeeds, Ticker Analytics and Sector Analytics are rebuilt from the full history every week. SKU- and product-level weekly metrics are processed incrementally: each week the current and previous calendar years are recomputed, and earlier years stay as last computed. Categories, brands and owners for those tables come from the product and SKU reference tables, which are rebuilt weekly, so classification is always current across the full history.

Published series are versioned. Each release of the Shelf Indices and brand headlines carries a version, a data-through date and a publication date. If a refresh changes previously published figures, the release says so in its revision note.

• Online listings: prices, promotions and ranges as shown on retailer and brand websites.
• Listed prices: the prices products are offered at, rather than transaction volumes.
• Covered retailers: figures describe the retailers Data Purl tracks.

Cite Data Purl as the source and scope the claim in the sentence itself: the retailers covered, that prices are online listed prices, the basis (for example matched-SKU or price per unit) and the date range. For example:

“Across Data Purl-covered US retailers, the matched-SKU price per unit of [category] changed by [X]% between [start] and [end]. Source: Data Purl (data-purl.com).”