How We Test AI Tools
The five weighted criteria behind every AIKERYX score, our hands-on testing workflow, how we verify pricing, and what we refuse to accept from vendors.
Last updated 2026-10-25
The short version
Every tool gets the same treatment: we pay for a real plan, run it on real work for two to four weeks, log what it consumes, and score it against five weighted criteria. Nothing is scored from a demo, a press release, or a vendor-supplied benchmark.
Scoring criteria
| Criterion | Weight | What we actually measure |
|---|---|---|
| Task completion | 30% | Does it finish the job we hired it for, on the first attempt or after iteration? |
| Output quality | 25% | Quality of results judged against a fixed reference task, not first impressions |
| Speed and reliability | 20% | Latency, failure rate, queueing, and consistency across repeated runs |
| True cost | 15% | Subscription price plus measured credit or token consumption per unit of output |
| Fit and learning curve | 10% | Time to first useful result, documentation quality, workflow integration |
Each criterion is scored from 1 to 5 in half-point steps, multiplied by its weight, and summed. The total is rounded to one decimal place. Scores are comparable within a category; they are not a universal verdict on quality.
The testing workflow
- Subscribe to a paid tier. We buy a plan comparable to the one our readers would buy. Free tiers are used only to describe what free users actually get.
- Run real tasks for two to four weeks. We use the tool on production-style work: assets we publish, video we ship, code we deploy.
- Log consumption. We record credits, tokens, or render minutes consumed, so cost claims come from measurement rather than from the pricing page.
- Cross-check against a competitor. The same task, with the same inputs and success criteria, is run on at least one alternative tool so results are comparable.
- Re-verify pricing. Immediately before publication we re-check the official pricing page and record the date of that check on the review.
- Re-review on a cycle. Categories are revisited periodically, and we re-test sooner when a vendor ships a significant change or a reader reports a discrepancy.
What we do not accept
- Payment for positive coverage, in any form.
- Vendor pre-approval of drafts, ratings, or conclusions.
- Editorial control attached to any sponsorship or affiliate arrangement.
- Review-only accounts that change behaviour. If a vendor supplies free or discounted access, we disclose it in the review and watch for differences from the retail experience. Where the test environment differs materially from what a paying customer receives, we say so explicitly.
How we verify pricing
We read the vendor's own pricing page and record the date we checked it, along with the plan and region the price applies to. Promotions, annual-discount pricing, regional differences, and applicable taxes are all set by the vendor and can change without notice, so the vendor's website remains the authoritative source. Where a figure is uncertain or varies by region, we say that instead of publishing a single number as fact.
Updates and corrections
Every review carries a last-updated date. When something material changes, we update the page and describe the change. When we get something wrong, we correct it and note the correction date so the history stays visible. Reader reports are a common trigger for both, and you can send them to hello@aikeryx.com.