THE THING WE BUILT:

ContextUIal

Twenty-two years ago, Minority Report promised us computers we could wave our hands at, screens that responded to gestures, and interfaces that just... understood what we wanted.

Instead, we got 47 different ways to copy and paste text between applications.

But what if your computer could actually see what you’re looking at and help you work with it? What if you could select any messy text, confusing interface, or random screenshot and get instant AI assistance that understands the visual context?

Turns out, the future is finally showing up. It’s just wearing regular clothes instead of chrome jumpsuits.

(Un?)fortunately no futuristic Power Gloves required.

And we built a thing.

ContextUIal in action (okay, not really, click the link)

AI That Meaningfully Interprets Your Screen

It’s a smart screen overlay that turns any part of your display into an AI-powered workspace.

1. Click & Drag to capture any area: messy text, complex interfaces, images that need descriptions.

2. Choose Your Mode:

   1. Fix typos (⌘⇧F) - Automatically corrects spelling and grammar, copies to clipboard.

   2. Follow instructions (⌘⇧C) - Sees instructions like "copy this as JSON format" and translates accordingly.

   3. Explain anything (⌘⇧D) - Describes what you’re looking at in a popup.

3. Get Instant Results - Watch messy content become clean and organized.

4. Copy or View - Results go to clipboard or appear in persistent popups

   
   

Notes from Building

1. Screen capture changes how you think about AI input. Most AI tools expect you to describe what you’re looking at, “I have this error message that says...” But when the AI can actually see your screen, you skip the translation step entirely. Instead of explaining the problem, you just show it. Way faster, way more accurate, and it works on stuff that’s hard to describe in words.

2. Prompt engineering for vision is different. Anyone can call a vision API, but structuring prompts for different visual use cases requires specific techniques. ContextUIal bakes in our prompt engineering best practices so typo fixing, instruction following, and content description each get optimized prompts without you needing to remember the differences.

3. Clear shortcuts are more effective than auto-detection. We started with a version that tried to guess what we wanted. It worked 98% of the time, but that 2% failure rate was maddening. Better UX means giving clear signals with simple shortcuts: ⌘⇧F for fix, ⌘⇧C for copy, ⌘⇧D for describe. It’s faster, sure—but more importantly, it makes the most of muscle memory. The difference between “I can do this quickly” and “I don’t even have to think” is everything. ⌘⇧Z to toggle, ESC to cancel, specific shortcuts for every mode. ContextUIal doesn’t get in the way. It just flows.

4. Visual feedback prevents confusion. The blue selection box shows exactly what area was captured, status messages tell you what’s happening, and result popups appear right where you selected. When you’re building screen capture tools, users need to see what the AI sees.

   
   

Built with ❤️ by Township

You’re receiving this because you signed up for Township’s “We Built a Thing” newsletter. We'll only send these when we, well, build things.