Skip to the page

Get in touch

Does a chatbot on your website increase conversions?

We followed a chatbot conversion figure on a vendor’s page back as far as it goes. Three company blogs later, it ended at a 2015 article about live chat with people, where it wasn’t an increase at all. It was the conversion rate of customers who chat before buying, credited to Forrester, with no link to a study.*

Journal · 6 minute read · Updated 17 November 2026 · Sources in the notes

The short answer

It beats silence, and that’s all the good evidence shows. In the best test so far, shoppers who got answers from an AI assistant bought more often than shoppers who got an automated note saying customer service was unavailable, and no more often than shoppers answered by people. We found no independent study showing that a chatbot on a website beats a clear page or a person who writes back, and the big number you were quoted probably came from a company that sells chatbots.

Strongest evidence first. Both experiments in the table have a control group: people picked at random who don’t get the thing being tested, so you can see what would have happened anyway. Each row is sourced below.

StudyWhat was comparedWhoResultWhat it doesn’t show
Fang and others, field experiment, 2026AI chatbot against an automated “unavailable” note, and against people44,614 shoppers, one online retailer, 202321.7% higher conversion than the note; no clear difference from peopleThat it beats a clear page, or suits a service business
Sun, Chen and Fan, 2021Live chat with sellers, in past sales data1,446 clothing sellers on Taobao, 2011–2012Chat went with higher conversionAnything about bots; not an experiment
Luo, Tong, Fang and Qu, field experiment, 2019Bot and human sales callers, bot disclosed or notMore than 6,200 customers, phone callsSaying “bot” first cut purchases by more than 79.7%Website chat today, or that hiding it is wise
Gartner surveys, 2024 and 2026What customers say5,728 and 3,566 customersMost want a way to a personWhat anyone did on a website
Vendor case studies and statisticsBefore and after, or chatters against everyone elseCustomers the vendor choseLarge lifts, which we don’t repeatThat the chatbot caused anything

What does the best experiment show?

That an AI assistant beats silence. In September and October 2023, a large cross-border online retailer took 44,614 shoppers who asked a question before buying and split them into two groups at random.1 One group got answers from a generative AI chatbot. The control group got the platform’s usual automated note “indicating that customer service was unavailable.”

The chatbot group bought at a rate 21.7% higher. The gap is statistically significant (one that size is very unlikely to be chance), and in plain numbers it’s about 1.3 more buyers for every 100 shoppers who asked. Answering people beats telling them nobody’s home. Hardly a shock, but now it’s measured.

The same paper put the chatbot up against people answering, and that result gets quoted less. It made no statistically significant difference to sales or conversion. In the authors’ words, it “matches the quality of human service but does not outperform it.” A chatbot that passed questions it couldn’t resolve to people brought in 11.5% more sales than people alone, which the authors call marginally significant; its difference in conversion could be chance.1 So the fair reading is that it did as well as a person, and better than nobody. That’s worth having if nobody is what your visitors get now.

Three things to hold it against. It was one retail platform, where many products had little written description: a long way from a business that sells a service. Three of the five authors consulted for or worked at the platform during the project, as the paper declares. And it began as a preprint (a paper shared before peer review), until the journal Management Science published it on 6 November 2026.2

What about live chat with a person?

It seems to help, and most where the page falls short. Sun, Chen and Fan looked at 1,446 women’s clothing sellers on Taobao, China’s leading consumer-to-consumer marketplace, from May 2011 to February 2012.3 Their results suggest live chat “has a positive impact on conversion,” stronger “when product information on web pages is less comprehensive.”4

Nobody was assigned to anything here: this was past sales data run through a statistical model, so it can’t rule out that sellers who chat well also do other things well. And people were doing the answering, with no bot in sight. One lesson still carries over: chat earns most where your pages leave questions open. Chatbot or live chat? weighs the two.

Does it matter that it’s a bot?

It did on the phone in 2019. Luo, Tong, Fang and Qu randomly assigned more than 6,200 customers to outbound sales calls from chatbots or human workers.5 Bots that kept quiet about being bots sold as well as proficient workers. Saying so before the conversation “reduces purchase rates by more than 79.7%.” Customers who knew saw the caller as less knowledgeable and less empathetic.

That was voice, outbound and seven years ago, a long way from a question typed into your website. And the lesson isn’t to hide the bot, because people want to be told: in Clutch’s 2025 consumer survey, 90% wanted brands to disclose when AI is used.6 A review in California Management Review says chatbots “should always clearly identify themselves as AI agents,”7 and Florida Bar Ethics Opinion 24-1 says a lawyer “must inform prospective clients that they are communicating with an AI program.”8 So a bot starts every conversation with less trust than a person. It earns it back by being useful, and by handing people to a person when it should.

How short is that trust? In Gartner’s survey of 5,728 customers in December 2023, 64% said they’d prefer companies didn’t use AI for customer service.9 In its 2026 survey of 3,566 customers, only 27% would try a chatbot again after a negative experience,10 and 87% said a way to reach a human agent is essential when companies use generative AI.11 That’s what people say in surveys, which can differ from what they do on a website. Read it as the mood in the room; it measures no conversions.

Where do the big chatbot numbers come from?

Mostly from companies that sell chatbots, or blogs that copy them. Double-digit lifts circulate widely, and so does a share of chatbot sales said to happen after hours. We couldn’t trace either to an independent study with a control group. The after-hours figures we did find counted inquiries or chats instead of sales, or came from a vendor’s write-up of a test on its own clients.

The trail we followed, from a vendor’s page back to 2015, and what the figure meant at each end.

None of that needs bad faith. Vendor numbers run high for ordinary reasons. A case study is the vendor’s best story, and the ones where nothing changed don’t get written up. “Before and after” has no control group, so it mixes in the season, your ads and whatever else changed that month. People who open a chat were already interested, so comparing them with everyone else, as the 2015 figure in our opening did, measures their interest and says little about the chat.

And the vendor decides what counts as a win. An email typed into the widget can count as a conversion whether or not that person ever buys. A chat opened can count as success, though it isn’t an inquiry, and the bot may take inquiries your form would have taken anyway.

Some vendors publish split tests of their own: a step up, though the seller ran them on its own clients and named no one who checked them. None of this makes a case study false. It shows a chatbot worked for someone. Whether it works for you is something you can test.

How can you tell if one would pay off for you?

Test it on your own visitors, against a group who don’t see it. That group is called a holdout, and it’s fair to ask a vendor for one.

  1. Pick the measure first. Inquiries and booked calls per visitor, and later how many became customers. Chats started don’t count.
  2. Show it to half. For a month, show the assistant to a random half of visitors, each kept in the same half on every visit. Both halves get the same weeks and the same ads.
  3. Set the end date before you start. Stopping the day the numbers look good finds differences that aren’t there.
  4. Compare inquiries, then their quality. Ten that fit beat twenty that don’t.

The catch is size. How many visitors you need depends on how many inquiries you get, and small counts swing by chance: if one half sends twelve inquiries in a month and the other nine, that tells you almost nothing. The retail experiment compared 44,736 shoppers against people answering, and the difference was still too small to tell from chance.1 With a few inquiries a week, a month can show a big effect and will miss a small one. So before the test starts, ask how many inquiries it needs before a difference means anything.

And when a vendor shows you its own numbers, ask:

  • Compared with what: no answer, a person, or the same site without the bot?
  • How were the case-study customers chosen?
  • What counted as a conversion: a chat, an email, an inquiry, a sale?
  • How many inquiries were in each group, and could the difference be chance?
  • Will you run a holdout on my site and show me the raw counts?

If the real gap is unanswered questions, a clearer page may close it before any bot does: see getting more inquiries from the site you have.

Questions

Is a “chatbot conversion rate” the same as my website’s conversion rate?

No. A chatbot conversion rate is usually the share of chats that end with contact details. Your site’s rate is inquiries divided by all visitors. The first looks far higher, because only interested people open a chat.

Can a chatbot lose me customers?

It can. In the same December 2023 Gartner survey, 53% said they’d consider switching to a competitor if they found out a company was going to use AI for customer service.9 A holdout catches this too: if the half with the assistant sends fewer inquiries, that’s your answer.

What if most of my questions come in after hours?

Then you’re closest to the retail experiment: an answer against no answer. Check the times on last month’s inquiries before you buy anything.

Where we come in

Oliver, the AI assistant on our site, is live, and we haven’t measured whether he brings in more or better inquiries. So we don’t claim he does. When we measure it, it will be against a holdout like the one above, and we’ll publish the result whichever way it goes. The choices behind him are in how we built our site assistant.

We build assistants for clients too, and when a clearer page would do more, that’s what we’ll recommend. We write back within a business day.

Where these numbers come from

We opened every source on 17 November 2026 and read the parts behind each figure, including the appendix tables of the retail experiment. Some publishers block automated reading, so we checked those pages through the Internet Archive’s recent copies and the abstracts the journals register with Crossref. We searched for an independent controlled study behind the vendor figures that circulate and didn’t find one.

  • Fang, Yuan, Zhang, Donati and Sarvary, “Generative AI and Sales Productivity: Field Experiments in Online Retail,” arXiv 2510.12049v6. Randomized field experiment; the preprint version. arxiv.org, read 17 November 2026.
  • The same study in Management Science, published online 6 November 2026. Peer-reviewed. doi.org, read 17 November 2026.
  • Sun, Chen and Fan, “Effect of Live Chat on Traffic-to-Sales Conversion,” Production and Operations Management, 2021. Observational study of existing data; peer-reviewed. doi.org, read 17 November 2026.
  • Lehigh University, “Chatting It Up,” fall 2021. University article describing the Sun, Chen and Fan data. business.lehigh.edu, read 17 November 2026.
  • Luo, Tong, Fang and Qu, “Machines vs. Humans,” Marketing Science, 2019. Randomized field experiment; peer-reviewed. doi.org, read 17 November 2026.
  • Gartner, press releases of 9 July 2024, 4 August 2026 and 2 September 2026. Customer surveys. gartner.com, read 17 November 2026.
  • Clutch, consumer survey press release, 14 October 2025. Survey; sample size not stated. clutch.co, read 17 November 2026.
  • Ren and Zhang, California Management Review, 28 April 2026. Review of published research. cmr.berkeley.edu, read 17 November 2026.
  • The Florida Bar, Ethics Opinion 24-1, 2024. Advisory ethics opinion. floridabar.org, read 17 November 2026.
  • ICMI, “The Stats Behind Chat: It’s Popular and Growing,” 2015. Trade article; the end of the trail in our opening. icmi.com, read 17 November 2026.