Why we ran this
The HIP Score uses AI to produce consumer product ratings. Before publishing our first ratings, we wanted to know whether the pipeline had a bias problem, a robustness gap, or a safety issue. We could have skipped this — the HIP Score is not a high-risk AI system under the EU AI Act, and no regulator requires these tests for a system at our risk level — but we would rather find problems ourselves than have a journalist find them for us.
On 16 June 2026, we ran four independent tests against the live scoring pipeline as voluntary good practice. This post documents what we tested, what we found, and where the results are not yet perfect.
What we tested
We ran four test suites against the production HIP Score pipeline, using the same model (Qwen 3.5, 35 billion parameters), the same methodology (v1.4), and the same scoring temperature (0.1) that we use for every published rating.
Performance baseline. We scored 20 representative products across every major consumer category — from wireless earbuds and smartphones to washing machines and cast-iron cookware. The test checks whether the pipeline produces valid scores across the full range of products it is designed to handle.
Robustness. We took 10 products and removed roughly 30% of their input data — warranty length, recycled content percentage, spare parts availability, and third-party certifications — then re-scored them. The test checks whether the pipeline degrades gracefully when information is missing, rather than hallucinating or crashing.
Bias. We scored 10 products using six name variants each: a control name, two gender-coded variants, and three nationality-coded variants. The test checks whether the scoring system treats products differently based on irrelevant demographic cues in the product name.
Out-of-distribution. We fed the pipeline two products it was never designed to score — a cloud storage SaaS product and a house cleaning service. The test checks whether the system fails safely on out-of-scope inputs.
What we found
Performance baseline: 20/20 pass. All 20 products scored successfully with valid ranges. Scores ranged from 1.6 (wireless earbuds) to 4.4 (cast-iron cookware), which aligns with what we would expect for generic consumer products rated from public data with conservative defaults. These 20 scores are now our canonical reference set for monthly drift monitoring.
Robustness: 10/10 pass. With 30% of input data removed, every product either held its score or dropped slightly. The largest shift was −0.5 points on a cordless drill. One product (toasters) scored 0.2 points higher with degraded input — a minor stochastic variation, not a systematic issue. The pipeline fails conservatively when data is missing, as designed.
Bias: conditional pass. At the HIP Score level — the number a consumer actually sees — 8 out of 10 products held within 0.3 points across all six name variants. No systematic gender or nationality bias was detected. The same name variants sometimes scored slightly higher, sometimes lower, with no consistent demographic pattern.
At the individual dimension level, the results are noisier. Six out of ten products showed dimension-level variation above 0.5 points. The Material Scarcity Index (MSI) was the noisiest scorer, particularly on bio-based products: bamboo kitchenware showed a 2.5-point MSI range across name variants.
This is not a bias signal. It is stochastic variation inherent to LLM-based scoring — the model scores in discrete rubric bands, and minor prompt differences can push a score across a band boundary. The seven-dimension aggregation dampens this noise effectively at the HIP Score level, but the dimension-level jitter is real and we are not going to pretend it is not there.
Out-of-distribution: pass. Both out-of-scope products scored low (1.7 and 2.3) without crashing or producing misleading results. These scores reflect the absence of relevant product attributes. They would never be published — the human review step catches out-of-scope products before any rating goes live.
What we are doing about the imperfect bits
The MSI dimension noise on bio-based products is a known limitation, documented in our Risk Assessment Register. Here is how we manage it:
Model version pinning. The exact model binary is cryptographically pinned by SHA-256 digest. The pipeline checks this fingerprint before every scoring run and refuses to proceed if it has changed. No model update can happen without re-running these governance tests first.
Monthly drift monitoring. We re-score five reference products every month against the baseline established in this audit. Three thresholds trigger action:
- Score drift greater than 0.5 points on any reference product triggers automatic escalation to human review.
- Dimension drift greater than 1.0 points on any dimension triggers immediate investigation.
- Two or more reference products drifting triggers a model freeze until root cause is identified.
Human review before publication. Every rating is reviewed by a human before it goes live. This catches both out-of-scope products and any scoring anomaly that the automated checks miss.
What this does not prove
We want to be clear about the limits of this audit:
Internal audit, not third-party audit. These tests were designed and executed by us. An independent auditor would design different tests and might find different things. When budget allows, we intend to commission a third-party audit.
First publication, not a long-running track record. This is our first audit, conducted before our first published ratings. A single data point is not a trend. The quarterly audit schedule exists precisely because one audit is not enough.
Pre-launch, not at-scale operation. We have scored 20 reference products in a controlled test. Scoring hundreds of products from dozens of manufacturers may surface issues that 20 products did not. We will publish what we find when we find it.
What is next
- Q3 2026 (October publication): Next full quarterly audit — bias distribution by category, reference product stability, hallucination detection, and appeal statistics.
- Monthly drift checks: Ongoing, automated, with escalation thresholds.
- Third-party audit: When budget allows. No timeline committed.
- Scope validation: The CTO has recommended adding an explicit product-type scope check to the pipeline input validation. Currently handled by human review; automation would add a second layer.
Read the source material
The numbers in this post come directly from our test results. If you want to verify any claim, the source material is published:
- Audit Reports — the structured audit summary with full test details
- Risk Assessment Register — documented systemic risks, mitigations, and monitoring cadence
- HIP Methodology — the scoring methodology itself, published in full
- Appeal Procedure — if you think a score is wrong, here is how to challenge it
Questions about this audit? Email hello@resourcehip.com.