Writing Voice-permeable interfaces
Your app already has a voice interface. It just can't hear yet.
You don't need a new interface or a chat tab. The screens you already have can learn to listen. We've started calling that voice-permeable.
Jon Yoder · · 4 min read
Part 1
Watch this
My wife has put up with a lot of bugs in the budgeting app we’re building. But there’s one thing she shows her friends. She opens a blank receipt, says “watch this,” presses a button, and talks.
What strikes me is what happens next. Nothing new. No chat window, no assistant with a personality. A pretty traditional screen fills itself in, and if a field is wrong, you tap it and fix it.
That’s what we mean by voice-permeable. Voice soaks into the interface you already have, and the interface stays what it was.
Try it
The blank receipt from the story above.
We start with a receipt. Then you can try the same idea in an issue tracker, a clinic chart, a gradebook, or a field service checklist.
Press and tap what you would say.
Nothing is recorded.
Nobody had to learn a new screen. Voice just became another way to fill in the form you already trust.
Part 2
One sentence, many rows
Filling in a blank is the easy part. The tedious part of most software is going down a list. A receipt with eight items is eight small decisions, and most of them are the same decision.
So the interesting sentence isn’t a command. It has an exception in it, the way people really talk. An exception is clumsy with checkboxes and easy out loud.
Try it, then tap a row and change it yourself. The list you change by voice is the same list you change by hand. There is one copy of it, so voice and touch can’t disagree, and the next sentence simply works from wherever you left things.
The demos are scripted, with no microphone and no AI model behind them: tap a phrase and the page plays out what that sentence would do.
Try it
Yesterday's run to Corner Market: eight items, two still without a category.
Not your world? Part 4 has the same idea in an issue tracker, a clinic chart, a gradebook, or a field service checklist.
Press and tap a phrase. Nothing changes until you accept. Tap any row to change it yourself, then ask again.
Nothing is recorded.
Part 3
When it hears you wrong
It will mishear you, and when it does, the cost is one tap. It shows what it understood on the rows themselves, before anything changes. The guess can vary from one try to the next. The screen and the checking don’t.
People already know what to do with that. Writing about speech and pen systems in 1999, Sharon Oviatt noted that “when a recognition error does occur, users alternate input modes in a way that tends to resolve it effectively.” Give someone a second way in and they’ll reach for it when the first one fails. A chat transcript makes that hard, because the result is a sentence you can’t touch. A screen makes it the obvious next move.
Try it
The same receipt, but this time it mishears you. Each phrase comes back with one word wrong.
Press , tap a phrase, then read what it heard. A wrong guess costs one tap.
Nothing is recorded.
The guess can be wrong. The screen is still the screen.
Part 4
Now try your world
Nothing here is special to receipts. Pick the kind of product you work on. It’s the same button and the same idea, on a very different screen.
This time it’s the whole app, a form and a list, with one button on both. Switch screens and the button comes with you, ready for whatever that screen needs.
These products are made up, so if you do this work for a living you’ll spot what I got wrong.
Your product
Pick a product, switch screens, press , and try the phrases.
Nothing is recorded.Try the app's other tab too. It has its own phrases.
Part 5
Where this comes from
None of this is a brand new idea. Richard Bolt paired speech and pointing in 1980, and researchers have studied it ever since under the name multimodal interaction. Oviatt’s “Ten Myths” still reads like a brief for designers. People like having both, but they mix them: in one study only 20 percent of commands combined two modes, and when they did, the two signals often arrived one after the other instead of together. Her evidence comes from pens and maps, not phones and language models, so treat it as a hint. The hint is that voice doesn’t have to do everything. It has to hand off well.
What I can’t find is much of this in the products people use every day. That’s the gap the word voice-permeable is for.
It isn’t for everything. If one tap will do, tap. And talking takes attention: Ben Shneiderman warned in 2000 that “since speaking consumes precious cognitive resources, it is difficult to solve problems at the same time.” So voice earns its place on the dull, predictable and repetitive work, where you already know the answer and resent entering it.
It’s also not the only future on offer. Julian Lehr argues for one AI layer that spans all your tools, called up by voice, and some people expect your app to move inside an assistant. This is a different bet: the listening moves into your app, onto the screens people already trust.
- Julian Lehr, The case against conversational interfaces ↗
- Yang et al., ReactGenie ↗
- Jakob Nielsen, AI: First new UI paradigm in 60 years ↗
- Richard Bolt, Put-That-There ↗
- Sharon Oviatt, Ten myths of multimodal interaction ↗
- Ben Shneiderman, The limits of speech recognition ↗
Tried this on your own product? I'd love to hear what happened.
New posts by email
One email when we publish something new, and when Keavi opens up. No tracking, unsubscribe in one click.