Free scoring matrix · score each vendor, then see the ranking

AI Vendor Evaluation Matrix

For procurement and technology leads comparing AI vendors. Score each vendor on identical criteria, adjust the weights to your case, and see how they rank and why.

Built by XONIK. Nothing you enter leaves this page.

Line illustration of vendors scored against criteria, with the cell that is never filled in, drawn as a grid with one cell failing.

In brief

This matrix helps you compare AI vendors fairly. You score each vendor on the same criteria, from fit to your task and where your data goes to exit and the cost of running, and you change the weights to match your case. It ranks the vendors, explains the leader against the runner-up and runs in your browser.

Score each from 0 to 5. Leave a name empty to leave that column out.

Criterion	Weight	Vendor 1: name	Vendor 2: name	Vendor 3: name	Vendor 4: name	Vendor 5: name
Fit to the task How to score this Judge it on your own cases, the awkward ones included, and not on a demonstration. Lowest score: it fails on your real cases, or you have not been able to try it on them. Middle score: it handles the usual cases and misses some awkward ones. Highest score: it handles your awkward cases correctly and you kept the results.	weight Fit to the task	Fit to the task, Vendor 1: score	Fit to the task, Vendor 2: score	Fit to the task, Vendor 3: score	Fit to the task, Vendor 4: score	Fit to the task, Vendor 5: score
Data handling and where data goes How to score this Lowest score: the vendor will not say where your data is stored or processed, or keeps the right to train on it. Middle score: location and use are stated, but only in a policy that the contract does not bind. Highest score: the contract says where data goes, which subprocessors see it, that it is not used for training without your written consent, and how it is deleted.	weight Data handling and where data goes	Data handling and where data goes, Vendor 1: score	Data handling and where data goes, Vendor 2: score	Data handling and where data goes, Vendor 3: score	Data handling and where data goes, Vendor 4: score	Data handling and where data goes, Vendor 5: score
Ability to inspect and export How to score this Can you see what it did on one case, and take your data and settings out? Lowest score: you see only the final answer, and nothing can be exported. Middle score: logs or step records exist but only on request, or export is limited to part of what you hold. Highest score: step records for each case, your settings and your data export in documented formats, and you control the access.	weight Ability to inspect and export	Ability to inspect and export, Vendor 1: score	Ability to inspect and export, Vendor 2: score	Ability to inspect and export, Vendor 3: score	Ability to inspect and export, Vendor 4: score	Ability to inspect and export, Vendor 5: score
Security evidence offered How to score this Judge what the vendor shows you before you sign, not what it says. Lowest score: a statement that it is secure and nothing more. Middle score: summaries of independent assurance are shared, but not the reports, or they cover a different product from the one you would use. Highest score: independent assurance reports, results of recent security testing and a named route for reporting a weakness are shared first, and they cover the product you would use.	weight Security evidence offered	Security evidence offered, Vendor 1: score	Security evidence offered, Vendor 2: score	Security evidence offered, Vendor 3: score	Security evidence offered, Vendor 4: score	Security evidence offered, Vendor 5: score
How change and model updates are announced How to score this Lowest score: models and terms change without notice and no version history is kept. Middle score: changes are announced after release, in release notes you have to look for. Highest score: changes, including model updates, are announced in advance in writing, you can test or pin a version before it reaches you, and the contract commits to the notice.	weight How change and model updates are announced	How change and model updates are announced, Vendor 1: score	How change and model updates are announced, Vendor 2: score	How change and model updates are announced, Vendor 3: score	How change and model updates are announced, Vendor 4: score	How change and model updates are announced, Vendor 5: score
Integration effort How to score this Score high when the effort is low. Lowest score: connecting to your systems needs custom work no one has scoped, and credentials with wide access. Middle score: standard connectors exist for some of your systems and the rest need work you have estimated. Highest score: it connects through documented interfaces with credentials limited to the task, and your own team has tried it.	weight Integration effort	Integration effort, Vendor 1: score	Integration effort, Vendor 2: score	Integration effort, Vendor 3: score	Integration effort, Vendor 4: score	Integration effort, Vendor 5: score
Support and escalation How to score this Lowest score: a shared inbox with no stated response time. Middle score: stated hours and response times, but no named contact and no route for a serious fault. Highest score: a named contact, response times set by severity, and an escalation route that reaches people who can change the product, all written in the contract.	weight Support and escalation	Support and escalation, Vendor 1: score	Support and escalation, Vendor 2: score	Support and escalation, Vendor 3: score	Support and escalation, Vendor 4: score	Support and escalation, Vendor 5: score
Exit and portability How to score this Lowest score: no way to take your data or configuration out, or exit terms that are silent. Middle score: data exports, but in a format only the vendor's tools read, or service stops the day you give notice. Highest score: your data, prompts, rules and configuration export in documented formats, the contract sets a period of continued access after notice, and you have tried an export.	weight Exit and portability	Exit and portability, Vendor 1: score	Exit and portability, Vendor 2: score	Exit and portability, Vendor 3: score	Exit and portability, Vendor 4: score	Exit and portability, Vendor 5: score
Documentation for your own obligations How to score this What you would need to show a customer, an auditor or a regulator. Lowest score: no instructions for use and no statement of intended purpose. Middle score: general documents exist, but not the logs, the fault-reporting route or the way to stop it that your role needs. Highest score: instructions for use, a stated purpose, access to the logs you control, a route to report a fault and a way to stop the system, all in writing before you sign.	weight Documentation for your own obligations	Documentation for your own obligations, Vendor 1: score	Documentation for your own obligations, Vendor 2: score	Documentation for your own obligations, Vendor 3: score	Documentation for your own obligations, Vendor 4: score	Documentation for your own obligations, Vendor 5: score
Total cost of running How to score this Score high for lower cost over time. Lowest score: usage charges can grow without a limit you set, and integration, review time and the cost of leaving are unknown. Middle score: usage charges are understood and the other costs are estimated roughly. Highest score: usage is capped or predictable, and you have counted integration, the people who run and review it, and the cost of leaving.	weight Total cost of running	Total cost of running, Vendor 1: score	Total cost of running, Vendor 2: score	Total cost of running, Vendor 3: score	Total cost of running, Vendor 4: score	Total cost of running, Vendor 5: score

How the ranking is produced

Name the vendors you are comparing and score each one on every criterion, from the lowest score to the highest score on the scale shown. A higher score is always better, so score high on integration effort and on cost when the effort or the cost is low. Under each criterion, the anchors describe the lowest score, a middle score and the highest score. Each criterion starts with a weight, and you can change any weight, including to zero to leave a criterion out. A vendor's weighted score is the sum of each weight times its score, divided by the sum of the weights, so the result stays on the same scale as your scores.

Vendors are ranked from highest to lowest, and equal scores share a rank. The leader is explained against the runner-up: the margin, the criteria on which it is ahead and the criteria on which the runner-up is ahead. For each vendor the result also names the criteria that added most to its score and the one that added least. A vendor left without a name is ignored, and a named vendor needs a score for every criterion. The starting weights are XONIK's judgment of what decides whether an AI system can be bought safely and lived with, drawn from the white paper Build, buy or agent. They are not drawn from a study of other buyers.

What this ranking cannot tell you

The ranking is arithmetic on your own judgments, so it is only as sound as the evidence behind each score. A score for fit means little unless you tried the vendor on your own cases, and a score for data handling means little unless you read the contract and not the sales page. Scores agreed in a room drift toward the most confident speaker, so have each person score alone first and compare afterwards.

A weighted average can hide a failure. A vendor can rank first and still have the lowest score on data handling, exit or documentation, and that is a problem to resolve before signing, not a trade for a good score elsewhere. A narrow gap between two vendors is not a decision. The tool holds no information about any vendor and recommends none. The ranking is not an audit, a certification or a benchmark. It cannot check whether a claim is true. What you enter stays on this page.

What a fair comparison of AI vendors takes, and what this matrix cannot supply

Vendor selections drift toward the best demonstration. A well-handled awkward question, a reference call the vendor arranged and a low quoted cost on the first page all leave a stronger impression than a clause about where data is processed. The matrix puts every vendor on the same criteria, with the same evidence expected for each.

It holds no information about any vendor and recommends none. It cannot check that a claim is true. It makes your own judgments visible and puts them in an order that can be argued over.

Criterion	Evidence to collect before you score it
Fit to the task	The vendor's results on your own awkward cases, kept in a file
Data handling	The contract clauses on location, subprocessors, training and deletion
Inspection and export	A record of the steps taken on one case, and an export you ran
Security evidence	The independent assurance reports and test results, not a summary
Change notice	The written commitment on notice, and the option to hold a version
Integration effort	A trial by your own engineers against one real system
Support and exit	Response times and escalation in the contract, and a test export
Documentation and cost	Instructions for use, the fault route, and a usage model on your volumes

Who scores, in what order, and how long does it take?

Collecting the evidence takes the time and runs through the whole evaluation. Scoring takes about a working session per vendor once the evidence is in the file. Ask the procurement lead, the security lead, the technology lead and the business owner of the task to score separately, then compare their sheets. Where scorers differ widely on one criterion, one of them has seen evidence the others have not.

Scoring as one group in a room drifts toward the most confident person. Separate scoring, then a short meeting to settle the differences, keeps the ranking sound.

How do you read the ranking, the margin and the weakest cell?

The ranking puts the vendors in order and explains the leader against the runner-up: the margin, where it is ahead and where the runner-up is ahead. A narrow margin is not a decision. The evidence has not yet separated the two, and the criteria where each leads are the questions to put to them next.

Look at the lowest score of the vendor that ranks first. A weighted average can hide a failure. A vendor with the best total and the lowest possible score on data handling, exit or documentation has a problem to resolve before signing. A good score elsewhere does not pay for it.

After the ranking, before the contract

01
Write to the leading vendors
List the criteria on which each scored low, and ask for answers in writing.
02
Move the answers into the contract
A promise made on a call is not a term. Have legal check that the data, notice and exit clauses match the sales material.
03
Score again with what came back
The second pass matters most, since it rests on evidence the vendors gave when asked.
04
Keep the sheet
Filed with the contract, the scores and weights show later why the choice was made and what was knowingly accepted.

When the matrix fits, and when it does not

It fits when

You have a shortlist and each vendor has been tried on your own cases
More than one function has a say, and the criteria must be agreed
An auditor or a customer may review the decision later

Look elsewhere when

You have not yet decided whether to buy; use the build, buy or agent decision tool first
You expect it to rank vendors from marketing material alone
One condition, such as a data location, should simply exclude vendors

A weighted matrix trades nuance for comparability. It is the right tool for putting vendors in order and the wrong one for finding the single fact that rules a vendor out.

What do people ask before using the AI vendor evaluation matrix?

Does the matrix recommend a vendor or check what vendors claim?

No. It holds no information about any vendor and recommends none, and it cannot check that a claim is true. It puts every vendor on the same criteria with the same evidence expected, and it makes your own judgments visible in an order that can be argued over.

Why agree the weights before scoring any vendor?

Weights set after the scores are in tend to move toward the vendor somebody already prefers. Fix them first with the sponsor and write down why each criterion counts as much as it does. Any weight can change, down to zero, but change it while no vendor is yet ahead.

Is the vendor with the highest total always the safe choice?

No, because a weighted average can hide a failure. A vendor with the best total and the lowest possible score on data handling, exit or documentation has a problem to resolve before signing, and a good score elsewhere does not offset it. A narrow margin is no decision.

Should we score the vendors together in one meeting?

Score separately first. Procurement, security, technology and the task's business owner each fill in a sheet, then meet to settle the differences. Scoring as one group drifts toward whoever sounds surest. Collect evidence beforehand, such as results on your own awkward cases and the contract clauses.

Why AI pilots stall before productionThe hand-off between the team that proved a model and the team that must run it.Report Technology EvaluationVendors and platforms compared on your criteria, fixed before anyone is demonstrated.Service AI Vendor Evaluation ScorecardA weighted scorecard for comparing AI vendors, with questions to ask in the meetings.Download Build vs Buy vs AI Agent Decision ToolBuild it, buy it, deploy an agent, or wait?Tool

Want a second opinion on your shortlist?

Describe what you are weighing in a few lines. A senior practitioner replies with a suggested first conversation.

Talk through your shortlist with us More free tools

AI Vendor Evaluation Matrix

What a fair comparison of AI vendors takes, and what this matrix cannot supply

Who scores, in what order, and how long does it take?

How do you read the ranking, the margin and the weakest cell?

After the ranking, before the contract

Write to the leading vendors

Move the answers into the contract

Score again with what came back

Keep the sheet

When the matrix fits, and when it does not

It fits when

Look elsewhere when

What do people ask before using the AI vendor evaluation matrix?

Want a second opinion on your shortlist?