Technology Research: the evidence before you commit
We establish what has been shown about a technology, by whom and under what conditions. Then we test the claims that carry your decision under conditions like yours, and report what holds and where it stops.
In brief
Technology research establishes what is known about a technology before you commit to it. We separate vendor claims from independent evidence, test the claims that carry the decision under conditions like yours, and report each finding with its confidence, its date and the developments that would change it.
On this page
Why does a convincing demonstration leave the decision open?
A vendor demonstration is built to succeed. The data is clean, the scenario is the one the product was tuned for, and the person at the keyboard has run it repeatedly. None of that is dishonest. The trouble starts when it becomes the only evidence a decision has, because the questions that decide whether you can live with a technology are the ones a demonstration is designed not to raise.
Those questions are about conditions: how it behaves on your oldest records, with the fields your teams leave blank; what happens when the security lead's rules switch off the feature the demonstration depended on; who in operations is paged when it degrades, and what they look at; whether it still behaves the same after the vendor's next release. They surface after the contract, the training and the dependent projects are in place, when reversing costs the most.
Technology research puts those questions first. We do not tell you whether a technology is good. We find out which of its claims have been shown independently, which have only been asserted, and which you can test cheaply enough to stop relying on anyone's word.
Not every source weighs the same
A technology arrives with a pile of material, and much of it is claim. We sort it before we test anything, because the sorting decides where the testing goes.
| Source | What it can and cannot establish | How we use it |
|---|---|---|
| Vendor documentation and demonstrations | What the product is designed to do and the limits the vendor admits; not how it behaves on data and rules other than the vendor's | Entered in the log as claims, each paired with the test that would confirm or refute it |
| Papers, standards and published benchmarks | Mechanisms, and results under stated conditions; not whether those conditions resemble yours, or whether anyone reproduced the result | Read for method before result; weighted by how close the conditions sit to yours |
| Incident reports, issue trackers and release notes | How it fails in other hands and how quickly the failure is fixed; not how often it fails | Mined for failure modes, which become test cases |
| People who run it today | What it costs in attention, the workarounds, what they would not do again; nothing outside their own setting | Asked the same questions every time; reported as accounts, never as measurements |
| Our own bounded tests | Behavior on your data and rules, in the cases we chose; not future use, or cases we did not try | Run to a protocol written beforehand |
From a question to a dated finding
- 01
Fix the decision and the question behind it
We sit with the sponsor and with whoever will own the technology afterward, and write the decision in a sentence: commit, run a bounded pilot, wait, or stop. Then comes the question that evidence could answer, and the result that would change the decision. What exists at the end is a question brief the sponsor has signed.
- 02
Build the evidence log
Everything read is graded as above, and vendor claims go in separately, each with its confirming test.
- 03
Write the protocol, then run it
The security lead and the data owner review the protocol for what we may touch and where it may run. The experiments are small and aimed at the claims that bear the heaviest weight, and they include the cases a vendor would not choose: ugly inputs, rules at their limit, failure and recovery. The protocol is dated before the first run, and the results include the runs that did not work.
- 04
Write the findings with the confidence on each
Every finding states what was shown, how it was shown, how far it generalizes and what would overturn it. A decision summary is written for the finance partner or board member who will not read the rest.
A pattern we are called into
An operations head asked to approve an assistant for the front line
The head of customer operations at a telecoms operator has been told that a language-model assistant can take over first-line triage of fault reports. The demonstration was convincing. The security lead has not seen the data flows, and finance holds a savings estimate from the vendor's own deck.
- The question is narrowed. The decision becomes whether triage can be assisted without more wrongly routed faults, measured on the operator's own ticket history.
- The evidence log shows where it tends to fail. Practitioner accounts and issue trackers agree that assistants do well on tickets that describe a symptom and badly on tickets typed in shorthand by field engineers.
- The protocol includes both kinds. It adds tickets with missing fields, drawn from a masked extract the data owner approves, and states beforehand what level of misrouting would end the idea.
- The result splits. The assistant holds on one class of ticket and does not on the other. The vendor's savings estimate had assumed it would hold on all of them.
Where it endsThe operator goes ahead with a bounded pilot on the class that held, with a monitoring threshold for misrouted faults agreed with the head of operations, and with a list of what to retest when the vendor ships its next major release.
A technology that is not ready is still a finding. It carries a date and the conditions that would reopen it.
What reaches the sponsor's desk
The findings report opens with the answer to the question in the brief, in a paragraph, and then goes finding by finding. Each finding carries a plain label for how well it is supported: shown under conditions close to yours, shown elsewhere, reported by practitioners, or asserted by the vendor and not tested. A reader who disagrees can open the evidence log and see which source bore the weight.
The decision summary is for the people who will not read the report. It lists the options open to you, what each depends on and what being wrong would cost in each direction. It also names the releases, rulings and vendor changes after which a finding should be rechecked.
The protocol and raw results go to your engineers, who can rerun the tests when the next version arrives, with us or without us.
When research should come before a purchase
Research costs time at the start, before anything visibly happens. Sometimes that time is well spent and sometimes it is not.
It fits when
- The decision is still open and independent evidence on the technology is thin
- One group inside the company is for it and another against, and neither can prove its case
- The technology will touch regulated data, so an early wrong commitment is hard to withdraw
- You can name the person who will own it if the answer is yes
Look elsewhere when
- The shortlist is already named and the choice is between vendors, which is Technology Evaluation
- The technology is mature in your sector, where a reference call and a pilot will teach you more
- Nobody with authority will act on the answer, whatever it turns out to be
- The real question is about a market, which Market Intelligence fits better
What you give up is the chance to say you moved early. What you get back is a position that survives the first hard question from finance or security.
Where does the research stop?
Bounded experiments show how a technology behaves in the cases we chose. They do not show every later use, and no test stands in for the first full cycle of real use. Where your data cannot leave your environment, we test on a close substitute and say how far it can be trusted.
- We need a sponsor and a data owner who can answer questions quickly while the protocol is written.
- We do not promise that a technology will behave in production as it did in test.
- We supply evidence for a business case and do not write it.
- When the work turns into choosing among named vendors, we say so and scope it as Technology Evaluation.
Where the method is written down
Client findings stay private, so what we can show is how we reach them. The pilots report examines the gap between a demonstration and daily use, and the matrix and scorecard hold evidence against criteria.
What buyers ask before commissioning technology research
What counts as evidence in your research?
Primary documentation, published technical work, recorded incidents, direct testing and the accounts of practitioners who operate the technology. A vendor's claim counts as a claim to be tested. Where only a claim exists, the report says so.
Can you test a technology with our own data?
Yes, where your rules allow it, and we will work inside your environment if they do not. When the data cannot leave, we design the test around a close substitute and state how far the substitute can be trusted.
Will you tell us if a technology is not ready?
Yes, and with the reasons. A finding that a technology is not ready is one of the most useful outcomes, because it spares you a commitment. The report names the conditions that would change the answer.
How long do the findings stay valid?
They describe the technology on the date of the work, and the report carries that date. We list the developments that would make a finding stale, so you know what to watch and when to ask for an update.
Who has to be involved on our side?
A sponsor who owns the decision, the person who would run the technology if you adopt it, and someone from security or data governance who approves what we touch. They are consulted at the start and at the findings.
Related
What does the evidence say about the technology you are weighing?
Describe what you are weighing in a few lines. A senior practitioner replies with a suggested first conversation.