AI Vendor Evaluation Scorecard
A scorecard for comparing AI vendors on the same weighted criteria, with a column for the evidence behind each score and a gate for the must-haves.
No email address needed. Download it and use it as it is.
What the scorecard and its question sheet hold
- Weighted criteria for fit to the use case, quality of output on your own data, data handling and privacy, security and access control, fit with your systems, transparency and control, support and continuity, and total cost over time
- A weight and a must-have flag against each criterion, both yours to change
- Columns for vendors side by side, and a notes column for the evidence behind each score
- A weighted score, a rank and a must-have check for every vendor
- A Questions sheet for the meeting, from where your data goes and whether it trains a model to what happens to you if the vendor changes terms or stops trading
How to use it in a selection
- 01
Fix weights and must-haves before the first demonstration
Set them before you see any vendor. Set afterward, they drift toward the vendor you already like.
- 02
Test on your own material and write down what you saw
Where you can, score from your own examples and put the evidence beside the score. A score with no note is an impression.
- 03
Read the gate before the rank
A vendor that scores zero on a must-have is flagged and should not be chosen on its weighted score.
Limits
The scores are your judgments, and the workbook holds no information about any vendor. The gate trips only on a score of zero, so decide what a pass means for each must-have before you score. A close result is a prompt to test further. Nothing here is a vendor review, an audit or a certification.
On this page
How does a vendor meeting run with the sheet open?
- 01
Send ahead
Share the questions on data handling and security, so the right people attend.
- 02
Run your example
Ask the vendor to work through your hardest case, not theirs, and to say what it does when it does not know.
- 03
Write it down
Put the answer in the evidence column the same day.
- 04
Score across
Score every vendor on one criterion before moving to the next, so one impressive meeting does not color the rest.
- 05
Check the gate
Read the must-haves last, then the rank.
What comes after a shortlist?
Choosing between building, buying and using an agent is a different question. The Build vs Buy vs AI Agent Decision Tool and the Build-vs-Buy Business Case Spreadsheet take it. The AI Vendor Evaluation Matrix on this site is the guided form of this scorecard, and our technology evaluation work runs the tests on your own material when you would rather not.
What do people ask before using the vendor scorecard?
When should I set the weights and must-haves?
Set them before the first demonstration, before you have seen any vendor. Set afterward, they drift toward the vendor you already like. Both the weights and the must-have flags are yours to change, so agree them with the people who will use the result.
How do I score a vendor fairly after a demonstration?
Test on your own material where you can, and write down what you saw beside each score, because a score with no note is an impression. Then score every vendor on a criterion before moving to the next, so a single impressive meeting does not color the rest.
What does the must-have check mean for a vendor with a high score?
A vendor that scores zero on a must-have is flagged and should not be chosen on its weighted score, however high its rank. The gate trips only on a zero, so decide what a pass means for each must-have before you score.
Does the scorecard tell me which vendor is best?
No. The scores are your judgments, and the workbook holds no information about any vendor. A close result is a prompt to test further. If you would rather not run those tests yourself, our technology evaluation work runs them on your own material.
Comparing vendors for a real decision?
Describe what you are weighing in a few lines. A senior practitioner replies with a suggested first conversation.