AI chatbot beats government leaflet on HPV vaccine literacy but not on uptake decisions

A randomised clinical trial published in JAMA Network Open found that an AI-driven chatbot raised HPV vaccine literacy among female caregivers in Japan more than a standard government leaflet did, but did not change their decisions to vaccinate ¹. The trial enrolled caregivers with unvaccinated daughters aged 12 to 18 years through a national research panel ¹.

What the trial found

The study was a parallel-group randomised trial run from November 30 to December 27, 2024, with 2-week follow-up ¹. Participants were randomised 1:1 to the chatbot or to the standard government HPV vaccine leaflet ¹. Literacy was scored on 7 factual questions about HPV and HPV vaccination, giving a scale of 0 to 7 ¹. The primary outcome was literacy immediately after the intervention and again at 2 weeks, and the secondary outcome was the decision to vaccinate at 2 weeks ¹.

Of 848 randomised participants, 704 of 841 (83.7%) were included in the immediate modified intention-to-treat analysis, split 353 chatbot and 351 leaflet ¹. At 2 weeks the analysis included 477 of 841 (56.7%), with 237 in the chatbot arm and 240 in the leaflet arm ¹. Of 703 participants, 447 (63.6%) were aged 40 to 49 years and 472 (67.1%) had college or university education ¹.

Immediately after the intervention, adjusted literacy was higher with the chatbot, with an adjusted mean difference of 0.30 points on the 0 to 7 scale (95% CI, 0.18-0.43; P < .001) ¹. At 2 weeks, mean literacy rose by 0.40 (SD 1.21) points with the chatbot against 0.28 (SD 1.13) with the leaflet, and the adjusted mean difference was again 0.30 points (95% CI, 0.13-0.47; P < .001) ¹.

Vaccination decisions at 2 weeks did not differ. 102 of 253 caregivers (40.3%) in the chatbot arm and 103 of 260 (39.6%) in the leaflet arm decided to vaccinate, an adjusted odds ratio of 0.74 (95% CI, 0.41 to 1.33; P = .31) ¹.

What it means for the market

The trial gives health bodies and suppliers of patient-facing conversational AI randomised evidence on a narrow question: whether a chatbot informs people better than the printed material a government already issues. On that measure the chatbot came out ahead at both time points ¹. On the decision to vaccinate it showed no measurable short-term effect ¹.

That distinction matters for anyone commissioning such a tool. A gain in knowledge of 0.30 points on a 7-point scale did not translate into a different share of caregivers deciding to vaccinate within 2 weeks ¹. The authors conclude that conversational AI “may complement official materials and health care professional recommendations”, a supporting role rather than a replacement for them ¹. The abstract gives no data on cost, scale of deployment or which chatbot product was used, so the commercial case cannot be judged from it.

Caveats

The trial ran in one country, among female caregivers recruited through a research panel, and most participants had college or university education ¹. Follow-up lasted 2 weeks, and the 2-week analysis included 56.7% of participants against 83.7% for the immediate analysis ¹. The trial is registered on ClinicalTrials.gov as NCT06702423, and the decision to vaccinate was only a secondary outcome ¹.