We Built an AI Job Search Assistant. Here's What We Got Wrong First.
Tai is the assistant inside TailorFit. Tell it "I just did a technical screen at Stripe" and it identifies the right application, logs the touchpoint with the correct date, and moves on. It handles application creation, activity logging, profile updates, fit checks, and job-listing URLs you paste into chat. Getting there meant shipping things we're not proud of. This is the honest list.
The raw JSON leak
The most embarrassing one first. Tai works by having the model return structured output: a single payload containing both the conversational reply and, when relevant, an action for the app to execute. Something like a reply string plus a create_application object with the company, role, and stage. The client parses that payload, shows you the reply, and quietly executes the action.
In the first version, when the model's output didn't parse cleanly, the chat panel fell back to rendering whatever came back. Some early users asked Tai a question and received a wall of raw JSON: braces, field names, their own half-parsed application data, where a plain sentence should have been. The assistant we'd carefully given a warm, slightly dry personality was answering in machine format. Nothing was exposed that the user couldn't already see. It was their own data on their own screen. But it looked broken because it was broken.
The fix had three layers, because the real lesson was that we'd treated model output as trustworthy instead of as user input. First, tolerant parsing: strip code fences, recover from truncation, validate the payload against a schema before anything renders. Second, a hard separation between channels: the reply string is the only thing that can reach the chat surface, and actions are validated independently, so a malformed action can no longer take the conversation down with it. Third, a real fallback: if parsing still fails, you see a plain "I didn't get that, mind rephrasing?" and we log the failure to Sentry and our generations table instead of the chat panel. The model is a component that fails. Design for its failures like you would for any other dependency.
We made Tai too agreeable about acting
Early Tai executed whatever it inferred. Say "I applied to Google yesterday" and an application appeared. That felt useful until the inference was slightly off: the wrong stage, the wrong date, a duplicate of something you were already tracking. Silent actions plus imperfect inference means users quietly losing trust in their own tracker. For a product whose core promise is "your pipeline is accurate," that's fatal.
The fix was tiering actions by blast radius. Low-stakes writes like logging a touchpoint execute immediately. Anything that creates or changes meaningful state shows a confirmation card first, with the parsed details editable. Anything that costs credits requires an explicit approval. Less impressive in a demo. Far more trustworthy by week three.
Dates were wrong for anyone east of the server
"I applied today" logged as yesterday for users in timezones ahead of UTC, because the server resolved dates against its own clock. Small bug, deeply corrosive. A tracker with wrong dates trains you to stop trusting the tracker. The client now sends its local date with every message, and Tai anchors every "today," "yesterday," and "last Friday" to that. Obvious in hindsight, like most timezone bugs. The lesson: relative time is user context, not server context.
What we'd tell anyone building one of these
Three things. Parse model output like it's hostile, because sometimes it is. Make the assistant earn the right to act silently, one action tier at a time. And build your observability before launch, not after. Every Tai interaction logs to our generations table with a prompt version. That's the only reason we could see the shape of these failures instead of guessing from support emails. Tai is the most interesting thing we've built and still the least finished. More to get wrong soon.