I was using Claude with a ClickUp connector to update some minor wording across a set of templates my team had spent the previous week filling in. What I expected: a heading change, some adjusted instructional text, everything else intact. What actually happened: Claude regenerated every page from scratch. Empty templates. A week of experiment notes, gone.

The interface had asked my permission. I said yes. What it didn’t show me was what “making edits” actually meant. The output looked perfect because my own page had no content in it to begin with. I had no idea anything was wrong until a designer messaged me: “Hey, my page is empty. What happened?”

We got the work back through version history. But the incident stuck with me, because it raised the question I built my AI×Design Melbourne keynote around earlier this month: when AI does the work and a human signs the name, who actually did this?

In financial services, this isn’t a thought experiment. Mortgage brokers using AI to draft client notes, extract data from payslips, or identify suitable lenders are still legally accountable for the output. The AI helps write the note. The broker’s name goes on the application. If the AI misread a salary, the broker is the one who has to explain it.

The three frameworks below are how I think about designing for that gap.

Framework 1: Diagnose the failure mode

Most teams building AI products are asking the same question: how do we make the AI more accurate? I’d argue there’s a more useful diagnostic. When users misuse an AI tool, they tend to do it in one of three ways.

Over-trust is when users accept AI output without checking. The AI sounds confident, the formatting looks complete, nothing about the response signals that anything is wrong. My ClickUp incident was the small version. The Commonwealth Bank case earlier this year, where roughly a billion dollars in home loans were approved on the basis of AI-generated fake documents, was the catastrophic one. Same pattern: the output looked complete, so no one looked closer.

Under-trust is the opposite. Users have access to AI that could help them, but they don’t use it, even when it’s accurate. The industry’s standard response has been to show the AI’s reasoning. It hasn’t solved the problem.

Miscalibrated trust is the most dangerous of the three because it’s hardest to detect. Users trust the AI on the easy questions where they already know the answer, then fall back to slower manual methods on the hard questions where the AI could actually add the most value. We’ve seen this in our own user research: brokers say they’d trust the AI for simple policy questions, but for complex client scenarios they’d still phone the bank directly. Those complex scenarios are exactly where the AI adds the most value, and exactly where the broker’s trust is weakest.

The design question isn’t whether the AI is trustworthy. It’s whether the user can tell which parts of it are.

Framework 2: The Checkpoint Spectrum

The standard industry response to AI risk is to add approval steps. A confirmation modal. A checkbox. A review screen. Checkboxes aren’t bad design, but a checkpoint that works once a month becomes muscle memory once a day. The broker stops reading it and starts clearing it to get to the next thing.

The checkpoint type has to match two things: how risky the action is, and how often the user encounters it.

Low-stakes, high-frequency moments need awareness checkpoints. A small AI indicator next to a field. A different treatment for AI-generated text. Nothing that interrupts. We built one of these into MyNoteWriter, our AI note-drafting tool. Compliance originally wanted a mandatory checkbox on every note. I pushed back, because a checkbox on a daily task becomes invisible by the fifth use. What we built instead was always-visible disclosure text positioned directly next to the copy button. The broker can’t reach the copy button without seeing the disclosure. No popup, no interruption, just placement at the moment of highest attention.

Higher-stakes moments need review checkpoints that put the AI’s output next to its source. The user does the verification by reading, not by clicking through a modal.

Irreversible actions need hard stops. Submitting a loan application. Locking in a rate. Something the user has to do deliberately, not a popup to clear. My ClickUp incident was a hard-stop problem dressed up as an awareness checkpoint. The action was irreversible, but the prompt was generic.

The mistake isn’t using checkboxes. The mistake is using the same checkpoint design regardless of risk and frequency.

Visibility isn’t verifiability

Showing AI reasoning is not the same as giving users a way to verify it.

Most reasoning traces are high-level by design. They tell you how the AI approached a problem, not exactly what it extracted. An AI can walk you through a perfectly logical chain of thought and still have misread the number that the whole chain is built on. It can reference sources that don’t exist with the same confidence as real ones.

Showing reasoning builds familiarity. It doesn’t build the ability to check the work.

The design response is to give users a fast path back to the source — the original document, the exact figure, the specific clause — so they can verify in seconds. If a broker is challenged on a decision years later, “I checked the original source” is something they can stand behind. “The AI’s reasoning looked correct” isn’t.

Framework 3: Designing for Authorship

Every AI product we’ve built has the same shape. The AI does the work. A human puts their name on it. The gap between those two things is what the interface either bridges or doesn’t.

The framework I use to test whether an interface is closing that gap is three questions the user should be able to answer at any moment.

What did the AI do? Show the AI’s work distinctly from the user’s. A different background colour, an inline tag, anything that makes it scannable at a glance which parts came from the AI and which came from the user.

What did I approve? Show the user what they’re about to sign off on, before they do. A preview that surfaces the destructive or irreversible parts of the action. If I’d had this on the ClickUp connector, the message would have read “this will clear content on six pages” instead of “are you sure you want to continue?”.

What can I still change? Treat AI output as a draft, not a final answer. Keep it editable until the user has confidently acted on it.

These three questions are simple to write and harder to design for. But they’re the test I’d run on any AI product before it ships to a user whose name is going on the output.

The interface is where accountability lives

In high-stakes industries, the difference between a clean outcome and a catastrophic one usually lives in the interface. In whether the unverified AI response actually looks unverified. In whether a destructive action shows what’s about to be destroyed. In whether the human signing their name knows what they signed.

The model’s outputs are the experience. Its failures are the design’s failures. And someone, at the end of every chain, is putting their name on what the AI just did. The job is to make sure they know what they’re signing.

Alex McMahon is Head of Design at Loan Market Group, where she leads AI experience design for MyCRM Intelligence — an AI product suite for Australian mortgage brokers.

If your team is building AI into a regulated or high-stakes environment, I deliver this as a 25-minute talk for design, product, and engineering audiences. It covers the three frameworks above, the six-stage testing protocol we use before any AI product reaches our brokers, and the production case studies behind each one. Recently delivered at AI×Design 2026 Melbourne. To bring it to your company or event, get in touch at alexmcmahon.com.au.

With the Voluntary AI Safety Standard (VAISS) in 2024 and the October 2025 “AI6” Guidance for AI Adoption, the Australian government has provided practical guardrails that translate policy into user experience recommendations.

If you’re building AI products in Australia, especially in regulated industries like financial services, healthcare, or government, these guidelines are important to take seriously. I expect these frameworks will become baseline expectations.

Understanding the Framework: 8 Ethics Principles → 10 Safety Guardrails → 6 Adoption Practices

Let’s clarify the terminology:

The 8 AI Ethics Principles are the foundation; fairness, transparency, accountability. The “why.”

The Voluntary AI Safety Standard translated those into 10 operational guardrails. The “what.”

The AI6 Guidance for AI Adoption condensed those into 6 essential practices. The “how.”

All 3 align. AI6 is the most practical starting point for design teams.

The AI6: Six Essential Practices

1. Accountability and Governance – Clear ownership, processes, compliance strategies

2. Impact Planning – Identify risks and benefits; define fairness for your context

3. Data Quality and Security – Ensure data is fit-for-purpose and protected

4. Transparency and Explainability – Make AI decisions understandable

5. Testing and Monitoring – Continuously evaluate performance and safety

6. Human Oversight and Contestability – Enable meaningful human control

These map back to the 8 original principles, but they’re structured around what product teams actually need to do. I have gone one step further and incorporated them into solution designs.

Turning Practices into Design Decisions

1. Accountability & Governance

The Principle: Clear responsibility for AI outcomes with identifiable accountability.

The Design Action: Build accountability indicators into the UI. Show who’s responsible for decisions and provide clear escalation paths.

Example: When an AI flags at risk customers, clarify whether the account manager or team lead is accountable for acting on it and provide a path to document the decision.

Designer’s Gut Check: If this AI recommendation leads to a compliance issue, can we trace who was involved?

Anti pattern: Diffusing responsibility so no one feels accountable.

2. Fairness & Impact Planning

The Principle: AI should be inclusive and not discriminate unfairly.

The Design Action: As a team, ask: “How would this fail a user in regional Australia with limited connectivity?” or “Does our AI disadvantage certain demographics?”

Equal accuracy? Equal opportunity? These are different goals requiring different approaches.

Designer’s Gut Check: Have we tested with diverse user groups? Have we documented our trade offs?

Anti-pattern: Assuming fairness because the AI doesn’t see demographic data.

3. Privacy & Security

The Principle: Respect privacy rights and protect data throughout the AI lifecycle.

The Design Action: Show users exactly what data the AI is accessing and why.

Example: If an AI summarises customer interactions, make it visible which records the AI is accessing. Let users scope the data range rather than silently pulling everything.

Designer’s Gut Check: Could a user explain to their customer exactly what data your AI accessed? If not, this could be insufficient transparency.

Anti-pattern: Treating AI data access as “backend intelligence” users don’t need to understand.

4. Transparency & Explainability

The Principle: Users must know when they’re interacting with AI and understand how it reaches decisions.

The Design Action: Build attribution/source into every AI output. Don’t just show a summary, provide visual links to source material so users can verify the logic.

Example: When AI recommends, show which data points drove that recommendation. Display confidence levels. Make the reasoning path visible.

Designer’s Gut Check: Can users trace the AI's logic from input to output? Replace “Smart Insights” with “AI-Generated Analysis based on [Source X].”

Anti-pattern: Hiding AI behind vague language or treating it as “intelligence”.

5. Reliability, Safety & Testing

The Principle: AI must perform reliably and fail gracefully.

The Design Action: Design low confidence states that change the UI when the AI is uncertain. Add warning indicators or surface confidence scores.

Example: When AI extracts data, for example, from documents, if confidence is low, highlight uncertain fields and prompt human review rather than auto populating forms.

Designer’s Gut Check: What’s the “safe failure mode”?

Anti-pattern: Treating all AI outputs with equal visual weight regardless of confidence.

6. Contestability & Human Control (Practice 6)

The Principle: People affected by AI decisions must be able to challenge outcomes.

The Design Action: Design the override pattern as a first class interaction. Every significant AI output needs a clear path to human review (my personal top suggestion).

Example: If AI flags content as high risk, users need a one click path to request manual review, with context about why they’re challenging it.

Designer’s Gut-Check: Is there a clear, dignified path to challenge AI decisions? Or are users clicking through help docs?

Anti-pattern: Burying override options in settings or making contestation feel like filing a complaint.

7. Human-Centered Values & Wellbeing

The Principle: AI should respect human rights, diversity, autonomy, and create net positive outcomes.

The Design Action: These aren’t about specific UI patterns, they’re about your design process. Ensure diverse representation in teams. Consult affected communities. Map unintended consequences.

Designer’s Gut Check: Who’s missing from our process? Whose perspective haven’t we considered?

The “Voluntary” Reality

Australia hasn’t introduced standalone AI legislation. The government is relying on existing laws and sector regulators.

But “voluntary” doesn’t mean “optional”:

Sector regulators are already enforcing these. ASIC applies financial services laws to AI decisions. The eSafety Commissioner regulates AI generated content. The OAIC enforces privacy obligations on AI systems.

In regulated industries, compliance requirements apply equally to AI and human decisions.

Where to Go From Here

1. Audit your AI features against AI6 Document gaps. Prioritise based on risk; high stakes decisions need stronger guardrails than low stakes features.

2. Appoint an AI Governance Owner Ethics can’t be siloed. You need an accountable person in your company for AI outcomes across the product.

3. Use the DTA’s Impact Assessment Tool If building for government, the Digital Transformation Agency’s Algorithmic Impact Assessment is mandatory as of December 2025. This can be useful for the private sector too.

4. Document the trade offs When you prioritise speed over explainability, document it. When you choose one fairness definition over another, document why. This record keeping could become a core requirement (or save you in the future).

5. Prioritise high stakes/risk features first

• First: Features affecting financial outcomes, legal status, or service access

• Second: Features processing sensitive data or making automated decisions

• Third: Lower stakes user facing features

Final Thought

Teams that treat AI6 as a design consideration now will build products that earn trust and withstand scrutiny in the future.

These guardrails aren’t about limiting AI, they’re about ensuring when AI makes mistakes (and it will), your product fails safely. When users challenge decisions (and they will), there’s a clear path. When regulators ask questions (and they might), you and your team have answers.

If you design with these guardrails now, you’re not just protecting against future mandatory regulation, you’re building products that respect the people who use them.

If you found this article useful, hit the subscribe button. Ill continue to share more insights on AI in product design in Australia.