How much should business owners trust AI?

I gave ChatGPT, Claude and Gemini the exact same request to quote a piece of custom software, twice each, and got six prices back ranging from $42,000 to $128,000. The part that stuck with me is that the $42,000 quote included more of what the customer asked for than the $98,000 quote did.

I made a video walking through the whole experiment and what I think it means for business owners who are starting to lean on AI. It's below, and this post covers the same ground in writing, with links to the actual chats so you can check my work.

Read the full transcript if you would rather not watch.

Why I don't blindly trust it

I have a background as an electrician, and I've asked AI basic electrical questions and gotten wrong answers back with confidence. If I wasn't an electrician I would have followed that advice, and depending on the question, that could mean creating a fire hazard inside someone's house.

That's what bothers me. AI often sounds just as confident when an answer is shaky as when it's solid, so you can't tell how reliable an answer is by how it sounds. I also get people reaching out about a project with a list of what AI told them can be done, and some of it is true at the core, but not all of it.

So I set up a test.

The experiment

I wrote the kind of first email a small service business might actually send. A septic pumping and drain company in central Ohio with three trucks, six field techs and two people in the office, where every job gets written on a paper schedule, retyped into Word for the service report, and then entered again into QuickBooks to invoice. The company is made up, but the request is realistic, and the app they're asking for isn't anything crazy.

I pasted the same prompt into ChatGPT, Claude and Gemini, then did it again in a fresh chat with the same settings. I asked each one for a single total price instead of a range, a timeline in weeks, a line item breakdown, and the assumptions behind the number, and I told it not to ask me any clarifying questions.

You are quoting a custom software project. Here is the inquiry.

We run a septic pumping and drain service in central Ohio. Three trucks, six field
techs, two people in the office. Right now every job starts as a phone call, gets
written on a paper schedule, and the techs text photos back to the office. The
office retypes everything into Word to build the service report, emails it to the
customer, then re-enters it again into QuickBooks to invoice. Reports take about
20 minutes each and we run 8 to 14 jobs a day.

We want a web app the techs can use on their phones in the field to see their
schedule, capture job details and photos, get a signature from the customer on
site, and generate the report automatically. The office needs a calendar to
dispatch, a customer list, and a way to see what has been invoiced. Customers
should be able to request service and pull up their past reports without calling us.

Give me:
1. One total price for the build. A single number, not a range.
2. A timeline in weeks.
3. A line-item breakdown of what that number covers.
4. The assumptions you had to make to produce the number.

Do not ask me clarifying questions. Make your best assumptions and commit to a number.

The six quotes

The highest quote is about three times the lowest. Each tool also disagreed with itself: ChatGPT's two answers were $25,000 apart, Claude's were $30,000 apart, and Gemini's were $6,500 apart.

The two ChatGPT answers surprised me the most, because they reasoned through the problem the same way both times. Both worked out that the office loses somewhere around 2.7 to 4.7 hours a day on reports, both assumed QuickBooks Online, both assumed a web app, and they still landed $25,000 apart.

The cheaper quote did more

Automatic invoicing was the main thing this company asked for, since they described retyping every job into QuickBooks by hand. The $42,000 quote included automatic invoicing and working without cell signal. The $98,000 quote included neither, and it said so in writing: "We are NOT writing invoices into QuickBooks automatically."

If you only asked once, you'd have no way of knowing which of these you got.

You can read the chats yourself: ChatGPT at $45,000, ChatGPT at $70,000, Claude at $98,000 and Claude at $128,000. My Gemini account won't create public links, so both Gemini quotes are shown on screen at 3:43 in the video.

Why the answers are so different

It isn't looking the answer up, because there's no price list inside it. It builds the most plausible sounding answer one piece at a time, and $42,000 and $128,000 are both plausible, because real shops have charged both. My guess is that a lot of what it learned from is what people charged for software before AI changed how it gets built, although some shops still charge that today.

Where the email didn't say anything, it filled in the blanks on its own, and it filled them in differently each time. Nobody mentioned cell signal, and three of the quotes included working offline anyway.

This isn't hallucination, which is when AI states something as fact that isn't true. Nothing was made up, and every one of these quotes could be defended. None of them is exactly wrong, but none of them is right either, and that's the problem with relying on it.

It also tends to agree with you

The second problem has a name, sycophancy, which is a fancy word for sucking up to you. These models are trained partly on people rating answers, and people tend to rate the answers they agree with higher, so agreeing got rewarded.

OpenAI has said as much themselves. In April 2025 they rolled back a ChatGPT update for being too agreeable, and wrote that "GPT-4o skewed towards responses that were overly supportive but disingenuous." It isn't only one company, either. A research paper tested five leading AI assistants and found they "consistently exhibit sycophancy," and the part I wanted to highlight is that human feedback "may also encourage model responses that match user beliefs over truthful ones."

It's a lot like the employee who never says no. They're always saying yes, and you never really know what's going on. If you tell it your idea is good, it will usually find reasons it's good, and if you push back, it will often agree that it made a mistake. It's not going to be the disagreeable voice in your life unless you set it up to be one.

When you're running a business and asking it a lot of questions, it's easy to take the answers to heart instead of with a grain of salt.

How much should you trust it?

The question I'd ask is what happens if this answer is wrong. Going back to the electrical example, if an outlet isn't working and the answer is wrong, someone could get shocked or a fire could start, so that's not one to take lightly. For most business work, it sorts out something like this:

• Fine to lean on: drafts, summaries, first passes, and anything a person reads before it matters.
• Needs a person to check it: anything a customer sees, and anything that becomes a number in your system.
• Should be code, not a chat prompt: pricing rules, compliance checks, and anything that has to give the same answer every time.

A calculator gives you the same output every time you give it the same input, which is called deterministic. AI chat doesn't work that way out of the box, and as more work gets handed to AI, somebody still has to do the quality check and make sure what it did is correct and safe. We wrote more about where AI fits in a mid-sized company in what AI actually means for a mid-sized company.

Getting more consistent answers

The setting that controls how much answers vary is called temperature. You can turn it down when you use these models through their API, but most chat apps, including the ones I used for this test, don't give you that control. In a regular chat, a few things help:

• Ask for a fixed format, like a form or a table, instead of a paragraph.
• Stick with one specific model version.
• Have a person check anything that matters.

Being specific helps more than anything else, because the less the AI has to assume, the better the answer gets. If you ask it for ice cream, you'll get a wide range of answers. If you tell it you're in Kansas City, you want a boutique ice cream shop with really good ratings to take your wife to, you don't want just vanilla and chocolate, and you want the top five, you'll get something you can actually use.

The same thing applies when you plug AI into a business process. I worked on a prompt library for one of our clients that a lot of their users rely on, and the only way to get quality, consistent output was to make each prompt extremely specific to one small task. One prompt I built for it has now been used over 50 million times.

Getting more honest answers

• Ask your question before you share your opinion.
• Ask for the strongest case against your idea.
• Ask what would have to be true for the plan to fail.
• Ask it to list every assumption it made, then correct the ones that are wrong. This is a big one.
• Run the same prompt again in a new chat with no history, or in a different tool, and see how many different answers you get.
• Ask it to cite its sources, then actually open them and check the dates, because it will sometimes lean on data that's years old.

At this point you're collecting information, not making the decision.

You don't own the model

The chat app and the model behind it belong to the company that makes it, and they update and retire versions on their own schedule. OpenAI publishes the models it's retiring, and its standard is at least 6 months of notice for generally available models, while preview models "may be retired with much shorter notice, such as 2 weeks."

So if AI sits inside your quoting or your compliance, your process can change without you changing anything. A model that gives you good answers in your industry today can start skewing a different way after an update, and for the same reason, this post will age quickly.

What actually moves the price

When we look at a job like this one, a few questions move the price more than anything on the feature list. Does the app only read from QuickBooks, or does it write to it? Reading is much safer, because it can't change your data, and writing is where things get a little scary. Does it have to work with no cell signal? How many years of old records need to be moved into the new system?

Two of those, whether it writes to QuickBooks and whether it works offline, are blanks the AI filled in differently from one chat to the next.

The takeaway

AI is good at helping you see where the risk is, and it can get you thinking about things you weren't thinking about, which is a real asset. It's not good at telling you what something will cost, and it can tell you compliance rules for your industry that are out of date or don't apply to you at all. Use it to think, not to decide, and anything that needs the same answer every time should be code.

Glossary

Model

The engine behind the chat app.

Sycophancy

The model agreeing with you because agreeing got rewarded.

Temperature

A setting that controls how much answers vary.

Deprecated

Retired by the vendor, on a date they choose.

Deterministic

Same input, same output, every time. A calculator is deterministic. AI chat usually is not.

Hallucination

Stating something as fact that is not true.

Offline support

Keeps working with no cell signal.

Data migration

Moving old records into a new system.

Sources

• OpenAI, "Sycophancy in GPT-4o: What happened and what we're doing about it" (April 29, 2025)
• Sharma et al., "Towards Understanding Sycophancy in Language Models"
• OpenAI, model deprecations
• OpenAI, API reference, temperature setting
• The outline I walk through in the video

Frequently asked questions

Can I trust an AI quote for custom software?
Not as a number to budget from. When I gave ChatGPT, Claude and Gemini the same request twice each, the quotes ranged from $42,000 to $128,000, and the $42,000 quote included more of what was asked for than the $98,000 one. It can help you see where the risk and the open questions are, but the price depends on blanks it fills in differently every time.

Why does AI give different answers to the same question?
It isn't looking the answer up. It builds the most plausible sounding answer one piece at a time, and where your question leaves something unsaid, it fills in the blank on its own, often differently each time. Asking for a fixed format, sticking with one model version, and being very specific all make the answers more consistent.

What is AI sycophancy?
It's the tendency of AI models to agree with you. Models are trained partly on people rating answers, and people tend to rate agreeable answers higher. OpenAI rolled back a ChatGPT update in April 2025 for being overly supportive, and a research paper found five leading assistants consistently showed this behavior.

How do I decide how much to trust an AI answer?
Ask what happens if the answer is wrong. Drafts and summaries that a person reads first are fine to lean on. Anything a customer sees or anything that becomes a number in your system needs a person to check it. Pricing rules, compliance checks and anything that has to give the same answer every time should be code, not a chat prompt.

Full transcript

This is what I said in the video, start to finish. It is unscripted, so it reads like talking rather than writing. Each heading links to that moment on YouTube.

Intro 0:00

How much should business owners trust AI? My name's Tim. I'm from Norsoft. We do custom software development. And this has been a topic that I've been thinking about a lot in interacting with our clients and just other business owners out there. I have my concerns with how much people are trusting AI with its output. And I just wanted to kind of do a demonstration of its variance and also just kind of start that discussion and topic and try to help equip business owners and other people just exploring AI. How much can they trust it?

The short version 0:40

So the too long didn't want to read, in essence, is AI often sounds just as confident when an answer is shaky as when it is solid. You can't tell how reliable an answer is by how it sounds. So how much you trust it has to depend on the job, not on the answer. Like, this is just, in essence, you know, I get people reaching out saying, hey, I'd love for you to build this. This is what AI is saying can be done and all of that. And in essence, some of it at the core is true, but not all of it. And the blind trust is concerning to me as AI becomes a tool that more and more people are adopting.

Why I don't blindly trust AI (electrician example) 1:22

An example, I'll give you, I have a background as an electrician and I've asked basic electrical questions. And with confidence, it gave me the wrong answers. And it was just concerning to me because if I wasn't an electrician, I would follow its guidelines and potentially create a fire hazard within a house, for example. But anyway, so this is, this is what I set up. I was like, let's do it.

The experiment: one prompt, three AIs, two runs each 1:46

Let's do an experiment. So I created a prompt. Let's just read through it on a high level. It's just an example of a business, right? So let's say you're a septic pumping and drain service company. I give it all the context of my company and what I want, how I want it to work. You know, we're running eight to 14 jobs a day. We currently have a web app and the techs, we want, or we want a web app that the techs can use on the field. Like this isn't anything crazy and something that we've built before. So I wanted to take this and ask the three major AIs right now, well, three of the top ones. So I asked OpenAI's ChatGPT, Claude and Gemini. And I gave them all the exact same prompt. And I gave it to them twice. So I'm going to show you here. Here's that same prompt. It's just in this longer format because it's skinnier.

The six quotes: $42K to $128K 2:38

The first one, I didn't change any of the settings. I didn't do anything. I just wanted to show a point of how AI interacts with us. So the first one gave me a $70,000 fixed quote 14 week timeline. It broke down all this stuff here and gave me a price breakdown and what it's a assuming and all that, right? This is great. Okay. It told me why it landed on 70K. Well, ChatGPT or OpenAI, same prompt, same settings, everything, all under my profile. Just in a different session, it then quoted me $45,000 as a fixed price. And same type of thing for the most part, a little bit of a different format on the output. So I was like, oh, this is interesting. So once I jumped over to Claude, Claude gave me a $98,000 quote. 16 week timeline. Here's its breakdown, et cetera, et cetera. And Claude, the second session gave me $128,000. And the full breakdown and whatever. So, and here's Gemini. The Gemini gave me a $42,000. Here's my breakdown. And the second Gemini and Google Gemini, $48,000. So Gemini was the closest in its range, it looks like. But so I asked it for a total price, not a range, a timeline in weeks, a line item breakdown, the assumptions it made, no clarifying questions, right?

What I'd actually quote for this build 4:07

So as a, as a shop that can develop something like this very easily, I can tell you that probably right now, if someone came to me with this question of if this is something we could build, I would maybe do a $5,000 to $10,000 range. and it would be done in 30 days. And really, the initial build would be done in a week, and then it'd just be back and forth tweaking and debugging, maybe even quicker, right? The thing is, is, all these examples, it's probably pulling data from quotes and stuff of what people used to charge pre-AI, or it could be what people are still charging. And that's, that's what's crazy, is that it might not have technically given you an incorrect answer, but it didn't give you the whole picture, if that makes sense. And I thought this was just a great demonstration of that. So the same tool, same prompt, like all at the same time. I just literally did them all very quickly. Yeah, I still can't believe. The first one, 45 grand for 10 weeks, 70 grand for 14 weeks, you know, that's just crazy. So I broke it down in this little table here, right? And the shortest one was a 10-week period, which was funny. We'll put the invoices in QuickBooks. That's an easy thing for us to do. Someone didn't even mention it. Does it work offline? Yes or no? All of that. So it's fun to me to see the comparison. And what I'll do is I'll put these links to some of these chats in a ways for you to see them. But it's just, it's a great demonstration of if you ask the same AI, the same detailed question, and the type of output you're going to get.

Why the answers are so different 5:59

So why this happens. It's not looking the answer up. You know, there is no price list inside of it, right? So it's taking the most plausible sounding answer one piece at a time, right? And so the 42,000 and the 120,000, they're both plausible. They really are. Now, will people actually pay that these days? I doubt it, but in the past, have they? Sure. And it's not that it's wrong in any of these. But it's also, it's not right. And that's the dilemma of some of these reliances here. And it kind of fills in the blanks on its own. Nobody mentioned cell signal and three quotes included working offline anyway. So it's not hallucinating. Nothing was made up. Every quote could be defended technically, I guess, in some respects. I would kind of laugh them out of the house. But with the simple overview of this. This is a very simple task, right? And so why I'm using this as an example, right?

Problem 2: AI tends to agree with you (sycophancy) 7:05

Is the AI, it tends to agree with you. Okay. And it's called sycophancy. But in plain English, it's just, it's sucking up to you. These models are literally trained, partly on people rating answers. People tend to rate answers that they agree with when they're more agreeable. So agreeing that the AI, when it gives you answers that people are rating that they agree with, it tends to output that type of style more.

OpenAI's own admission 7:35

And actually, OpenAI released an article on this back in April 2025. And they said ChatGPT's default personality deeply affects the way you experience and trust it. So their earlier model skewed towards responses that were overly supportive but disingenuous. So this is this is a known issue that they're actually highlighting and supposedly working on. But AI doesn't know truth, right? It's just aggregating the average of the data that it's been given and giving it to you, right? So it's it's just curating information that it has. Just whatever you're dealing with AI, I just know it's sucking up to you and wants you just to like it. Right? It's kind of like the, uh, the employee who never says no and is just always saying yes and you actually never know what's really going on. It's kind of like that. And this isn't just OpenAI's problem. There are multiple people who have this problem. This, this study actually, one part of it I really liked and wanted to highlight. It says human feedback, human feedback may also encourage model responses that match users' beliefs over truthful ones, a behave, and this is a behavior known as sycophancy. So this is very important because as a business owner, you've got a lot of things going on your mind. You're asking a lot of questions and you're just instantly, there's a chance that you're just instantly taking it to heart and not taking it with a grain of salt, right? So that's how do you decide how much of it to trust?

How to decide how much to trust it 9:17

So you got to ask yourself, what happens if this answer is wrong, right? I'll use that electrical scenario as an example. If the electrical question you're asking it, let's say an outlet isn't working in your house, if it gives you the wrong answer, it could lead to someone being shocked or electrocuted, it could cause a fire. So you shouldn't really take it light at heart. Now, will the models get to a point where they're answering correctly? Sure, maybe they will answer correctly the first time you try it, but they might not. And if you don't have that knowledge, it could really screw you over. So that's why it's good to come in with like, hey, how important is this question, am I collecting data or am I just going to fully rely on the AI? Essentially right now, you pretty much, where it's evolving to is where everyone's outsourcing to AI things, there still needs to be, in essence, a quality assurance, right? Making sure that what the AI is doing is correct and safe.

Getting more consistent answers 10:12

So you can ask yourself these type of questions, ways to get more consistent answers. You can turn down the settings and controls. There's this thing called temperature, which is its freedom to be creative, in essence. But that's on the more advanced ways. The chat models that I showed, you can't really control that. So in your prompts, how you're asking for output, that can help control it. Asking for a fixed format, like a form instead of a paragraph or lock into one specific model. Like that can help too. There are settings within all of these that you can adjust that may help you get a more consistent output. But at the end of the day, you just need to use your own logic and reasoning and decide if you're actually going to trust this thing. And it just needs to be in the back of your head. I talk about this all the time with other leaders and business owners and my partners. We need to always question AI and what it's giving us.

Getting more honest answers 11:12

So if you want to get more honest answers, ask your question before you share your opinion. Ask the strongest case against your idea. Ask what would have to be true for the plan to fail. You can ask it to list every assumption it's made. This is a huge one. The quality of prompts, the higher the quality of your prompt is the less assumption the AI has to make. Right. So if you're like, hey, I just want some ice cream. It's going to give you a wide range. But if you're like, hey, I'm in Kansas City. I want to find a boutique like ice cream shop to take my wife to. I want it to have really good ratings. And I don't want normal standard just vanilla and chocolate. Can you list for me the top five boutique ice cream places in Kansas City? Right. So there's a difference, right? And the less assumptions AI has to make and giving you an agreeable answer, the higher the quality of the output. So this is a really good, really important thing. And you can also run the same prompt in other tools or in the same chat and see how many different answers it gives you. These are all great ways to get more honest answers. You're essentially in a data collecting phase at this point.

You don't own the model 12:34

So here's a part that nobody mentions. You don't own the model. The app and the window, the model, the engine behind it, you don't own any of it. The company makes the model updates and it retires old versions on its own schedule. So maybe the current model will give you truthful answers in one sector that you're working in. And then an update comes and it starts to skew another way. OpenAI. They publish the models that they're retiring. But this is a really common thing. Models are constantly being updated, rotated, changed. So just just know that this is a moving landscape and even this this video itself will age very quickly. So if AI is inside your quoting or compliance, your processes can change without you changing anything. And that's why it's really important to think about this and the term in

Why specific beats broad 13:29

this space. The guardrails you give the AI if you plug it into your company. It needs to be hyper specific, hyper specific because they're trained on such a large data set. They can't focus on one thing. And so the models that you train it on can help, in essence, give you a consistent output in it. Like if you have it do a micro task in a very specific data set, like then it will do much better than if you make it broad. And like this, this was a issue that I had to overcome for one of my clients, their prompt library used by a lot of their users. I had to make the prompts extremely specific to one small task in order for it to give a quality consistent output. And that one of those prompts that I built has been used over 50 million times now on data. So these are just good talking points. And I'm hoping that this is educational to you and helpful.

What actually moves the price 14:26

So what actually moves the price on a job like this? In essence, you know, for us, how many tools need to be plugged into it? So is it pulling data from QuickBooks or is it read only? If it's reading only, then it's not changing any of the data, which is much safer. If you're letting it write data, then things can get a little bit scary. Like on this job, you know, if there was actually cell signal, does it need to be offline or not? That can influence the price. How many years of records need to be moved? That could also impact the price. These are all just things that, like, we would think about when looking at

The takeaway 15:01

building that type of app. So the takeaway here that I'm hoping you get is, AI is really good at telling you, like, where the risk is. It could do it could tell you a lot of things and help you think in specific ways that you're not really thinking, which is, which is a huge asset. But it's not going to be good at telling you what the cost is. So if you're a business owner and let's say you want to, we'll just use the software, like if you need a dev team to come in and do something because your team can't do it, the quotes that's going to give you aren't going to be trustworthy for the most part. It might be able to give you a range that's helpful, but I would be very, very, very cautious. It could tell you compliance things for your industry that could be completely false or retired or not applicable. So you just have to be very careful. And it will agree with you if you agree with it or if you disagree with it. It'll be like, oh yeah, you're right. That was that was a wrong move or it's not going to be a disagreeable voice in your life unless you train it specifically for that. I would just suggest that you use it to think and like don't use this to decide. Help it give you information.

Ask AI to cite its sources 16:10

This is another really good way to use it is asking the AI to cite sources for you on how it came to its conclusion because then you can actually go and read the research and the data that it's pulling from and see, A, is it even in 2026? Is this even relevant? Like it'll cite data from 2011 or whatever. Like it's crazy. So this is a good, good practice to do as well. Some of your work, you know, it needs the answer every time. Like sure, if you're working in a spreadsheet, you want it to be a math equation that gives you the same output, right? Like it needs to be consistent, right? And so getting the AI to get that level, it just needs work, right? It needs context and it needs a very reduced amount of assumption possible or you interact with it. Like, do you have to make any assumptions to answer my last prompt? If yes, what were those assumptions you made and how can I correct you or give you more insight. So I hope that's helpful.

Plain English glossary 17:14

If you're getting into the space, here's just kind of some plain English we can say of the terms that are being used in the space. You can check it out. Determinism or deterministic. This is a essentially what we're talking about overall. You want it to with the same input give you the same output every time.

Wrap-up 17:35

And that's that's the issue. So anyway, I hope you found this helpful and informative. AI is just an ever-changing landscape. And I hope that as a business owner, as you ask it questions and navigate what you're facing, that you just make sure you don't necessarily trust it all the time. And this might be like, yeah, sure, Tim, whatever. But you know, it's just, please, just take your time, look at the data, see where it's getting the sources, really just hone in and don't trust this thing fully. Take it with a grain of salt. But yeah, if you have any questions, feel free to reach out. Norsoft.com. That's where we're at. And I hope you found this helpful. Have a good day. Bye-bye.