I recently read an essay by Alan Kay from 1989 that originally featured in The Art of Human-Computer Interface Design, edited by Brenda Laurel. This was essentially before the internet, before smartphones, and long before any of the AI assistants we now use daily. And yet it describes the exact problem we’re still trying to solve.
Kay makes a distinction I haven’t seen articulated as clearly anywhere else. Humans have extended themselves in two ways throughout history.
First, through tools. Physical things we manipulate directly. A hammer, a keyboard. The feedback is immediate. You hit a nail and it moves.
The second way is through management. Convincing other entities to work toward our goals. Other people, historically. But increasingly, software that acts on our behalf. What Kay calls agents.
The interface challenge for these two categories is completely different. With tools, the question is how efficiently can I manipulate this? With agents, the question is how do I know if I can trust this to complete the task I set?
This has been something that I’ve been thinking about this as I’ve played with various AI products. The onboarding for poke.com felt immediately familiar. Within minutes it felt like it “knew” me. But after a week the novelty wore off. Sure, it drafted emails, but I always felt the need to adjust before sending.
This is essentially the gap Kay identified. We’re trying to apply tool-based expectations to something that requires a completely different interaction pattern.
Kay wrote that the thing we most want to know about an agent is not how powerful it is, but how trustable it is. The agent must explain itself well enough so that we have confidence it’s working for us rather than as what he calls an escaped genie.
He predicted agent development would move in two directions. First, expanding into domains where mistakes don’t matter much. Where undo is easy. These would move fast. The second direction would move slowly. Domains where undo is hard or impossible. Where mistakes affect real relationships or irreversible decisions.
Looking at where AI has actually expanded, this prediction holds remarkably well. Code completion moved fast. Autonomous decisions in healthcare or finance remain constrained. The pattern isn’t about technical capability. It’s about reversibility, confidence and trust. Not in the technical abilities, but in the agent itself.
What strikes me most is his claim about explanation. Kay argued that well-done explanation will be needed regardless of how the agent is instructed. The interface challenge isn’t about making AI more conversational. It’s about making the reasoning legible enough to calibrate trust. When I ask an AI to draft something and then need to adjust it before sending, that gap represents a trust calibration failure. The AI was confident. I wasn’t. And I couldn’t easily understand why our judgments differed.
The hardest part to accept is that this might not be primarily a technical problem. Tool-based interfaces can be evaluated through direct feedback. Agent-based interfaces require something closer to the trust calibration we use with human colleagues. But with humans, we have shared context. We have social structures that create accountability. We build trust through repeated interactions where we observe judgment against outcomes.
None of these mechanisms exist for AI agents. The conversational interface creates an illusion of familiarity, but the underlying trust architecture is still largely missing.
Kay saw this clearly in 1989. We’re still figuring it out. But we’ll get there.
