Technology Evaluation: choose with the criteria fixed first
We help you choose between technologies, vendors or approaches by fixing the criteria before anyone is demonstrated, then testing each candidate on your own scenarios.
In brief
Technology evaluation chooses between technologies, vendors or approaches on criteria fixed and dated before any vendor is demonstrated. We shortlist on evidence, run each candidate through your own scenarios, review contract, cost and exit risk, and recommend with the reasoning and the conditions that would change the answer.
On this page
Why does the favorite usually win before the evaluation starts?
Selections tend to open with demonstrations and a feature list. The vendors that present best set the terms of the comparison, and the weights are assigned afterward, once a favorite has emerged. Whatever the final score says, it was shaped by an answer somebody already wanted, and a sharp finance partner or an auditor can see that in a minute.
The questions that decide the outcome are rarely on the list: what the product does with your untidy data, what it costs to run once introductory terms end, how your data leaves if you leave, which team has to change how it works. These surface in the second year, when the choice can no longer be undone cheaply. A selection can be defended on the day and still be wrong, because the evidence it rested on was the vendor's.
Criteria weighted after the demonstrations measure the favorite, not the need.
So the weights are written, agreed and dated before any vendor is contacted.
Whose question is the choice answering?
A choice that is contested inside the company is usually a bundle of separate choices wearing one name. Each person who will rely on it is asking something different, and the criteria have to make room for every question that can later be used to reopen the decision.
| Who relies on the choice | What they will ask later | What the criteria and proof must already show |
|---|---|---|
| Security lead | Where does data go, who can see it, and what happens in a breach on the vendor's side | A must-have criterion on data handling, tested against your classification rules, not the vendor's summary |
| Finance partner | What does it cost in the year after the introductory terms, and what does leaving cost | A running-cost model with stated assumptions, and a costed exit |
| Head of operations | Who is paged, who retrains, and what slows down during cutover | Scenarios written by the people who do the work, scored by them |
| Product owner or sponsor | Does it do the job that justified the budget | A proof on that job, with the cases a demonstration skips |
| General counsel's office | Can we get out, and who owns what is produced inside it | Contract, data portability and ownership terms read against the criteria |
The order of work decides the answer
- 01
Criteria
Agreed with the people above, weighted and dated.
- 02
Shortlist
Candidates that fail a must-have drop out with the reason recorded.
- 03
Proof
The same scenarios, from your work, for every candidate.
- 04
Risk and cost
Contract, exit path and running cost, costed.
- 05
Recommendation
A memo that says what would change the answer.
How a proof on your own scenarios is set up
The scenarios come from the head of operations or the product owner, drawn from real work: a typical case, a hard case, and the case that went wrong last quarter. They are written down before any vendor sees them, and every candidate receives the same set. We run the proof in an environment we control, or in yours, so that no candidate tunes the setup for itself.
The people who will use the choice score the results, with the vendor's name hidden where the interface allows. If a vendor declines the proof conditions, the report records that, and the decline counts in the assessment. The cost to you is their time: someone in operations has to write scenarios and score them, and the security lead has to approve what data the proof touches.
What exists at the end is a scored set of results for each candidate, the scenarios themselves, and the moments where a candidate needed help from its own staff to complete a task.
A pattern we are called into
A retail group, two camps and one platform decision
A CIO at a retail group must choose a commerce and promotions platform. The engineering lead favors a flexible platform that the team assembles. Merchandising favors a packaged suite that runs the promotions calendar out of the box. Each camp has a vendor on its side, and both have already seen their favorite demonstrated.
- The criteria session moves the argument. Instead of debating products, the camps argue about weights, such as how quickly a merchandiser can change a promotion rule without a developer. That argument is settled and dated.
- The shortlist loses a candidate. One product fails a must-have on data residency, with the reason written down.
- The proof uses last season's hardest week. Both remaining candidates work through the same promotion changes, scored by merchandisers and by the engineering lead.
- The risk review finds the exit problem. The packaged suite exports data only in a format that needs rework, and its running cost after the introductory terms is well above the figure in the proposal.
Where it endsThe recommendation names the flexible platform with conditions on merchandiser tooling, and a sensitivity note on what would reverse it. Merchandising does not get its favorite, yet signs the memo as a fair statement of the criteria it helped set.
Evaluate, run a pilot, or renew what you have
A formal evaluation takes time from your senior people. It is worth it when the cost of the wrong choice is hard to reverse.
An evaluation fits when
- More than one candidate is credible and the choice is contested inside the company
- Leaving later would be expensive, because of data, retraining or dependent systems
- Finance, security or an auditor will ask how the choice was made
- The decision may be build, buy or an agent, and the options are not directly comparable
Something lighter is better when
- A regulation or an existing integration leaves one realistic candidate
- The choice is cheap to reverse, so a short pilot teaches more than a study
- You are not yet sure the technology works at all, which is Technology Research
- The decision has been taken and you want the paperwork to match
You give up some weeks and the comfort of an early decision. You gain a choice with a written basis, and one that each camp helped define.
What the recommendation memo contains
The memo opens with the recommendation and the conditions attached to it. A recommendation to choose one product with conditions on configuration and contract is more useful than an unqualified name. Behind it sit the dated criteria record, the scorecard for each candidate, the proof results with the scenarios, the cost model with its assumptions, and the contract review.
The sensitivity note says what would change the answer: usage growing well beyond the proof, a change in the vendor's terms at renewal, or a requirement the criteria did not anticipate. It tells you when to revisit the choice, and it gives your negotiators the points the risk review found. Your counsel drafts and negotiates; we say which terms to seek and which to refuse.
If we have any relationship with a candidate, we state it before the evaluation begins and step back from that vendor's part of the work.
Where the recommendation stops
A proof shows how a product behaved in the scenarios we ran, under the conditions we set. Daily use differs, and a product that did well in the proof may meet problems in the first full cycle of real use.
- The people who will rely on the choice must agree the criteria, which takes their time.
- Vendors must accept the same proof conditions. If one declines, the report records it.
- We recommend. We do not negotiate or sign for you unless the scope says so.
- We do not guarantee that the chosen product will perform in daily use as it did in the proof.
The tools this method is built on
The matrix and the proof scorecard are the working instruments for the criteria and the proof stages, and the perspective below explains why a pilot flatters what production must pay for.
What buyers ask before commissioning a technology evaluation
How do you stay independent of the vendors you evaluate?
We state any relationship with a candidate before the evaluation starts, and we step back from that vendor's part if there is one. The criteria are dated first, every candidate meets the same proof, and the record lets your team check that we followed it.
Can you evaluate approaches as well as products?
Yes. A selection is sometimes really a choice between building, buying and using an agent. We apply the same discipline to those: fixed criteria, a proof on your scenarios, and a written recommendation with its conditions.
What if two candidates score the same?
Then the criteria are not discriminating, and we say so. We look for what separates the candidates in the scenarios and the risk review, and if nothing does, we recommend on the cost of leaving, which is the cheaper one to be wrong about.
Do you help with the contract once we choose?
We give you the points that matter from the risk review: terms to seek, terms to refuse and exit provisions. Your counsel drafts and negotiates. If you want us at the table for the technical questions, the scope can say so.
Related
Were your selection criteria fixed before the demonstrations began?
Describe what you are weighing in a few lines. A senior practitioner replies with a suggested first conversation.