There’s a new kind of insurance agent working around the clock. It never sleeps. It never has a bad day. It answers every question with calm, fluent, authoritative language, and it is wrong with unsettling regularity.
Artificial intelligence has moved fast into the world of insurance. Consumers are using AI chatbots to compare coverage, understand policies, decide what they need, and ask what their policy covers after something goes wrong. The technology feels like a breakthrough, instant answers, no awkward phone calls, no waiting on hold.
But there is a dangerous flaw at the heart of using AI for insurance guidance, and it’s not a software bug that will be patched in the next update. It’s structural. It’s inherent to how these systems work. And it’s already costing people real money.
AI Doesn’t Know What It Doesn’t Know
The term in the industry is “hallucination.” It’s a clinical word for something alarming: AI language models sometimes generate information that is completely false, stated with total confidence, in fluent professional language, with no indication whatsoever that the answer is fabricated.
This isn’t a rare glitch. It’s a documented, ongoing characteristic of how large language models operate. The system is rewarded for sounding plausible, not for being accurate. When it doesn’t know something, it doesn’t say so. It fills in the gap with what sounds right.
In insurance, that’s a serious problem.
Consider what happens when a customer asks an AI chatbot whether their homeowners policy covers flood damage and the chatbot, confidently and helpfully, says yes. It doesn’t. Standard homeowners policies universally exclude flood. If that customer walks away believing they’re covered, they don’t buy a separate flood policy. They don’t think twice about it. Until the water comes.
Or consider a small business owner who asks an AI tool whether their business insurance covers a data breach. Most standard business policies don’t. Cyber coverage is typically a separate policy, and New York’s SHIELD Act creates real legal and financial exposure for unprotected breaches. If the AI says yes, the owner believes it, skips the cyber coverage, and finds out the truth at the worst possible moment.
These aren’t hypothetical scenarios. Insurance industry researchers have documented AI systems inventing coverage limits that don’t exist, misapplying state regulations, citing exclusions that aren’t in the actual policy, and generating claim summaries that assign causes of loss never documented. What makes it particularly dangerous is that the errors look like competence. The language is fluent. The tone is confident. Someone without insurance expertise has no way to recognize that the answer is wrong.
A Court Already Weighed In
The liability question around AI advice isn’t theoretical anymore. There’s a legal precedent worth knowing.
In early 2024, a man named Jake Moffatt lost his grandmother. He needed to fly home urgently and consulted Air Canada’s AI chatbot about bereavement fares. The chatbot told him he could purchase a full-fare ticket and apply for the discounted bereavement rate retroactively. He did exactly that, captured a screenshot of the exchange, and submitted his request. Air Canada denied it, citing its official policy.
Moffatt took Air Canada to the Civil Resolution Tribunal in British Columbia. Air Canada’s defense was remarkable: the company argued that the chatbot was essentially a separate legal entity, that the airline couldn’t be held responsible for what it said.
The tribunal rejected that argument. It found that Air Canada owed Moffatt a duty of care, that the chatbot had given him inaccurate information, and that Air Canada had “failed to exercise reasonable care to ensure the information’s accuracy.” Air Canada was ordered to pay.
The tribunal was direct: it makes no difference whether information comes from a static page or a chatbot. The company is responsible.
This ruling sent a clear signal that courts intend to hold companies accountable for the advice their AI tools deliver. It also revealed something important about consumer behavior: when AI gives an authoritative answer, people act on it. They don’t double-check it. They don’t verify it with an expert. They trust it, and why wouldn’t they? It’s right there, in professional language, from what appears to be a knowledgeable source.
In insurance, where acting on wrong information can leave you unprotected at the exact moment you need protection, that trust is dangerous.
The Disclaimer Nobody Reads
AI tools are not insurance advisors. They’re not licensed. They’re not regulated. They carry no professional accountability.
Look at the fine print on virtually any AI tool, whether it’s a general-purpose chatbot or an insurance-specific comparison tool. You will find language that says, in various forms: this output does not constitute professional advice and should not be relied upon for financial, legal, or insurance decisions.
The AI itself is telling you not to trust it for exactly this purpose.
That disclaimer isn’t a formality. It’s a legal shield, a way for the company behind the tool to distance itself from the consequences when the AI gets something wrong. And it will get something wrong. The question is whether you find out before or after something happens.
A licensed insurance agent operates under a duty of care to the client. When they give advice that turns out to be incorrect, there is accountability, regulatory, professional, and legal. There is someone who can be held responsible. There is recourse.
With AI, there’s a disclaimer.
What AI Simply Cannot Do
Setting aside the hallucination problem entirely, there are things AI cannot do when it comes to insurance. Not because the technology isn’t sophisticated enough. Because the work requires something the technology doesn’t have.
AI cannot read your situation. It knows what you tell it. It doesn’t know what you didn’t think to mention. It doesn’t know that you started running a home-based business last year and your homeowners policy may have a business exclusion that just became relevant. It doesn’t know that your teenager started driving and your auto policy hasn’t been updated. It doesn’t know that you’ve been renovating your house in stages and your dwelling coverage hasn’t kept pace with the rebuild cost. A relationship-based agent knows these things because they’ve been asking the right questions for years, and they remember the answers.
AI cannot advocate for you during a claim. When you file a claim, particularly a complex one, a disputed one, or one where the carrier’s initial response feels wrong, you need someone in your corner who knows your policy, knows the carrier, knows what’s reasonable, and is willing to push back on your behalf. A chatbot cannot make that call. It cannot write the letter. It cannot escalate to a supervisor. It cannot tell a claims adjuster, from experience, that a particular interpretation of the policy language doesn’t hold up. You are on your own.
AI cannot be accountable. If the advice you got from an AI turns out to be wrong, if you relied on it, made decisions based on it, and now find yourself underinsured at the moment of a loss, there is no one to hold responsible. The company will point to the disclaimer. The chatbot will not remember the conversation. You absorb the consequences.
AI cannot know your local market. Insurance isn’t a national commodity that works the same everywhere. Carriers write business aggressively in some zip codes and pull back from others. Coverage that’s standard in one part of New York may be harder to find or priced differently in another. An experienced local agent knows which carriers are stable, which ones are difficult at claims time, and which policies have language that looks good until you actually need it.
The Relationship Is the Point
For most people, the most important thing their insurance agent does has nothing to do with the policy itself.
It’s the conversation two years in, when you mention offhandedly that you’re thinking about buying a rental property, and your agent asks four follow-up questions you hadn’t thought about and realizes your umbrella policy needs to be restructured.
It’s the phone call when the tree falls on your house at midnight and there’s a person on the other end who knows you, knows your policy, and tells you exactly what to do and what not to do before the adjuster shows up.
It’s the annual review where someone who knows your life notices that your kids are grown, your mortgage is paid off, and your coverage situation looks nothing like it did when the policy was written.
It’s trust built over time, through real conversations, with someone who has professional skin in the game, someone who benefits when things go right for you and is accountable when something goes wrong.
No improvement in AI will replicate that, because the value isn’t in the information delivered. It’s in the judgment applied, the relationship maintained, and the advocacy offered when you need it most.
How We Use AI, and Where We Draw the Line
We’ll be honest: we do use AI. Just not the way you might think.
We use it the way a good accountant uses a calculator, as a tool that handles routine work faster and more accurately so the people in our office can focus on what actually requires a human being.
AI helps us analyze policy documents and flag coverage gaps before a renewal conversation, so when we sit down with you, we’re already prepared. It helps us compare carrier options across multiple companies quickly, so we can bring you the best fit rather than just the most available. It helps us stay current on regulatory changes, like updates to New York’s SHIELD Act or shifts in carrier appetite in specific zip codes, so our advice reflects what’s actually happening in the market. It helps with the administrative work that used to eat up hours: processing documents, tracking policy expiration dates, flagging accounts that need attention before a problem develops.
What AI does not do in our office is talk to clients. It does not answer coverage questions. It does not make recommendations. It does not replace a single conversation, a single phone call, or a single relationship. There are no chatbots on our website. No virtual assistants standing between you and a person. When you contact us, you reach us, real people, in a real office, who know your name.
We have been a locally owned, independent insurance agency for over 80 years. That’s not a marketing line. It’s eight decades of people in this community trusting us with their homes, their businesses, their cars, and their families, through recessions, through storms, through pandemics, through every kind of loss imaginable. That trust was not built by an algorithm. It was built one conversation at a time, by people who showed up and stayed.
We embrace tools that make us better at our jobs. But we have watched too many people come to us after relying on an AI assistant and discovering, at the worst possible time, that the confidence of the answer had nothing to do with the accuracy of the answer.
Insurance is not a transaction. It’s a promise. The only way to make sure that promise means something when it’s needed is to have a real person build it, explain it, maintain it, and stand behind it.
You Deserve a Person
The next time you’re tempted to ask an AI whether your coverage is adequate, whether your policy covers a particular situation, or whether there’s a gap you should know about, consider what you’re actually asking.
You’re asking about your financial security. Your home. Your business. Your family’s ability to recover when something devastating happens. You’re asking about one of the most consequential financial decisions in your life.
That question deserves a person who knows you, is accountable to you, and will still be there when the answer matters.
We’ve been that person for this community for more than 80 years. We intend to keep being exactly that.
AI can recommend a restaurant. Let a real agent protect what you’ve built.
We are a locally owned, independent insurance agency that has served New York individuals, families, and businesses for over 80 years. Every person who contacts us reaches a real human being, no bots, no virtual assistants, no automated gatekeepers. If you’d like to talk about your coverage, we’re here. Not a bot. Us.