method // how we measure
supereo asks AI engines the questions your buyers ask, judges every answer, and grades every change against questions the change couldn't touch. This page is how, with the numbers the product runs on.
Every tracked brand gets a fixed panel of buyer questions: the 20 its free audit wrote, 4 more written from what the audit learned about the brand, and 6 that name the brand (what it is, what it costs, how it compares, alternatives to it, reviews, and whether it fits): 30 to start, 40 at most. You confirm the panel and its competitors before it locks.
Which question is asked moves a number more than chance does, so the panel is versioned. Each question is a series of its own: adding, editing or removing one makes a new version, and only that question's series starts again. How often AI recommends you counts the questions that don't name you; the ones that do are reported beside it.
Every question runs on six engines, 3 times a week each, and never twice on one engine within 30 minutes. Each is asked the way a buyer reaches it, with nothing forced:
Every answer is read by supereo's judge, which makes three calls about your brand, and the same three about each competitor:
Recommended is the number supereo leads with. If you think the judge got an answer wrong, say so on it: the answer is judged again with your note, and the counts follow when the second reading agrees with you.
A rate is the share of the last 4 weeks of answers that recommend you, shown as answers out of every 100. Its range is where it likely sits, its 95% range: 1.96 standard errors either side. Only run-to-run luck within a question moves a fixed panel's rate by chance, so the range comes from each question's own spread, with half a hit added on each side so a question answered the same way every time still carries some.
Every change supereo ships gets a test, registered before the change goes live. So does a change you ship from supereo's recommendations, when your tracked questions have room for it: registered before you ship it, or as it ships, the moment you mark it shipped or supereo's weekly fetch of the page sees it. A test holds its questions until its verdict, so a change shipped while other tests hold too many of them goes ungraded, and supereo says so. Its rules are saved before any of its own runs, and nothing asked once the change is live counts as before it.
The rules are saved when the test is registered and never edited after: the questions, the engine, the bar, the window, the harm stop and the re-checks. A test whose change goes live before its baseline is done takes the tracker's own runs of the 28 days before; if it is still short, it can end only as Made it worse or Too close to call. A test whose change isn't live 28 days after it was registered ends, since its baseline would be too old.
From the first runs after its clock starts (below), the test shows where it is heading: the difference so far, with its range. It is read again as every run comes in, and a number read that often will cross any line by chance sooner or later. So early signal never uses a verdict word, and nothing is decided from it.
Once a test's clock has run 45 days and every question has its 25 runs since it started, supereo computes the verdict once and writes it, whatever it says. If the runs aren't in by day 60, it writes the verdict then from the answers it has. The size reads as answers, as in “10 more of every 100 answers recommend you”.
| Verdict | When |
|---|---|
| Worked | The whole 95% range is above zero, and the estimate is at least 10 more of every 100 answers. |
| No change | The whole range is below 10 more of every 100, and it includes zero. The change stays, unless you've set supereo to roll such changes back. |
| Made it worse | The whole range is below zero. The change is recorded to be undone: supereo reverts its own, and tells you about yours. |
| Too close to call | Anything else. The verdict says so instead of rounding up. |
A change that hurts shouldn't stay live for weeks. While a test runs, supereo looks at most every 84 hours, twice a week, once every question has 5 runs since its clock started. If the whole 99% range is below zero, a stricter bar than the verdict's, the test ends at once as Made it worse, and the change is recorded to be undone. The harm stop can only ever say Made it worse: nothing is called a win early.
Every change that worked is measured again every 28 days after its verdict: 10 runs of each of its target and control questions over 7 days, against the test's own baseline.
A test's change goes live when supereo's fetch of the page sees it, when you mark it shipped and the next fetch shows it, or when supereo's own pull request is served. Its window starts when AI first reads the changed page, as your Cloudflare or Vercel edge sees it: an AI company's own agent opening it for someone's question, never for one of supereo's own, so a read that comes while supereo asks that company a question whose answer pulled the page (within 2 minutes of the ask, or until its answer is back when it takes longer) doesn't start it. Runs between the change and that read count on neither side. With no edge connected that sees the page, or no such read within 7 days, the window starts when the change went live. The page is fetched daily while its test runs. An edit to it by anything but the change marks the test, which runs on with the mark on its record. Changes to the competitor pages its questions cite are noted on the test, never counted against it.
A verdict comes only from supereo's own question runs, because only they have controls. Visitors confirmed from AI and real users' reads of your pages are shown beside your numbers, and never counted in a verdict.
