Part 1 of a 3 Part Research Series
Why The Most Important Question In Artificial Intelligence Has Nothing To Do With Intelligence
The debate surrounding artificial intelligence has become strangely one-dimensional. Every week seems to bring another announcement of a larger model, a more powerful data centre, a new funding round, or a fresh prediction about how many jobs AI will eliminate over the coming decade. The assumption underpinning most of these discussions is rarely questioned. It is simply taken for granted that today’s limitations are temporary and that future versions of these systems will eventually become reliable enough to replace large portions of human knowledge work.
Perhaps they will, yet it is worth pausing for a moment and asking a question that surprisingly few investors appear willing to consider. What if the reliability problem proves far more difficult than the intelligence problem?
This distinction may ultimately determine whether artificial intelligence becomes the most profitable technology in history or merely another useful tool that changes the way people work without fundamentally replacing them.
Most users of AI systems are familiar with the phenomenon commonly referred to as hallucination. The term is unfortunate because it sounds like a minor software defect that engineers will eventually eliminate. In reality, it describes something much more fundamental. Large language models do not understand truth in the way humans understand truth. They are prediction engines. Their objective is to generate the most probable continuation of a sequence of information. Most of the time this produces remarkably useful results. Occasionally it produces complete nonsense. The difficulty is that the nonsense is often delivered with the same confidence and fluency as the correct answer. This creates a problem unlike almost anything we encounter in traditional software.
If a calculator occasionally produced incorrect answers, nobody would trust it. If accounting software occasionally invented transactions, it would never be deployed in a professional environment. If a navigation system occasionally guided drivers into a lake, it would disappear from the market almost immediately. Yet investors seem remarkably comfortable assuming that systems capable of confidently generating incorrect information will eventually replace lawyers, accountants, doctors, engineers and financial analysts. The economic implications of this assumption are profound.
Consider a lawyer using artificial intelligence to draft a complex contract. If the system produces a document that is ninety-five percent accurate, most observers will describe that as an extraordinary achievement. Yet from the perspective of the lawyer, the remaining five percent may contain enough risk to require a complete review of the entire document. Every clause still needs to be verified. Every reference still needs to be checked. Every conclusion still requires professional judgement. The lawyer becomes more productive, but the lawyer is not replaced. The same logic applies almost everywhere that accuracy matters.
Doctors may use AI to review scans more efficiently, but the final diagnosis remains their responsibility. Accountants may automate large portions of routine work, but somebody still signs the financial statements. Engineers may accelerate design processes, but somebody must ultimately certify that a bridge will not collapse. In each case artificial intelligence creates value without removing the need for human accountability.
This distinction is often overlooked because it is less exciting than the vision of fully autonomous systems. Yet it may be far more realistic.
The current investment narrative increasingly assumes that artificial intelligence will trigger a wave of labour displacement on a scale not seen since the Industrial Revolution. This assumption underpins forecasts of extraordinary productivity growth, widespread deflationary pressures and trillions of dollars of future economic value. But these forecasts depend on AI replacing workers, not merely assisting them.
If reliability constraints mean that humans remain embedded within the process, the economics begin to look very different.
A productivity enhancement of twenty or thirty percent is economically valuable. It may justify widespread adoption. It may improve corporate profitability. It may even alter the competitive landscape in certain industries. However, it is not equivalent to eliminating entire categories of employment.
History offers a useful perspective. The internet transformed the global economy, yet it did not eliminate the need for human judgement. Search engines made information more accessible, but they did not eliminate researchers. Spreadsheets made calculations easier, but they did not eliminate accountants. Email accelerated communication, but it did not eliminate management. Technology tends to augment human capability far more often than it replaces it entirely.
Artificial intelligence may ultimately follow the same pattern. If that proves to be the case, many of the assumptions currently embedded in financial markets deserve closer examination. The claim that AI will usher in a profoundly deflationary era becomes less certain. The expectation of widespread labour displacement becomes less convincing. The justification for unprecedented infrastructure spending becomes more difficult to defend.
None of this requires artificial intelligence to fail. In fact, AI may prove to be one of the most useful technologies ever developed. Businesses may adopt it enthusiastically. Productivity may improve substantially. Entire industries may be transformed. The crucial point is that usefulness and economic value are not the same thing.
Investors appear increasingly willing to assume that greater capability automatically translates into greater profitability. History suggests otherwise. Some of the most transformative technologies ever invented generated disappointing returns for the investors who financed them because the economic value created by the technology was far smaller than the expectations embedded in asset prices.
The future of artificial intelligence may ultimately depend less on how intelligent the models become and more on whether they can become genuinely trustworthy. Investors increasingly assume that these are different stages of the same journey. They may not be. The industry has already demonstrated that machines can produce convincing answers. Whether they can consistently produce correct ones may prove to be the far more difficult challenge—and the one upon which the entire economic case for AI ultimately rests.
