Blog

Introduction

Background

Patient experience is the general satisfaction a patient obtains during the process of receiving health care services []. In particular, patient experience has been considered as one of the central pillars of health care quality [,]. There are currently various developed questionnaires or scales to measure the patient experience in different health care settings, such as the Outpatient Experiences Questionnaire [,], the Chinese Patient Experience Questionnaire [] and the Picker scale []. Previous research pointed out that patient experience was closely related to the quality of health care delivery, involving outcomes such as patient safety and clinical effectiveness [,]. Some qualitative studies further indicated that a more positive patients’ communication experience with their physicians is related to higher general satisfaction with the quality of health care delivery []. Thus, policymakers worldwide increasingly prefer using patient-experience data over performance indicators to evaluate the quality of health care services.

In recent years, the Chinese government has been dedicated to improving the patient experience [] and encouraged the development and application of artificial intelligence in the health services in various scenarios (eg, conversational agents [], artificial intelligence-assisted diagnosis [] and decision-making []). This initiative promotes the construction of smart hospitals with the aim of solving the urgent and difficult problems when people seek health services, thereby continuously improving the quality of health services.

Conversational agents are artificial intelligence programs (ie, chatbots) that engage in dialogs with patients via the mobile devices before a consultation with their visit physicians in outpatient departments []. With the contextual question-answering agents, the patients’ information on their conditions, symptoms, and past medical history (eg, disease history, examination or medication or operation history, allergy history, family history, and personal history of drinking or smoking) can be collected and then sent to their physicians’ workstations in a structured form []. Some review studies have pointed out that artificial intelligence-assisted conversational agents had the potential to save time by reducing the time required for history taking and improve consultation efficiency, thereby resulting in high levels of satisfaction [,]. And these studies further indicated there were few quantitative studies that evaluate the above-mentioned effects or outcomes of conversational agents with objective measures [,].

Objectives

Under the national policies on the digital transformation of the health care industry, artificial intelligence-assisted conversational agents have begun applying to enhance the health care delivery in tertiary public hospitals in economically developed regions of China (eg, Shanghai in 2021) []. The artificial intelligence-assisted conversational agents have not only helped patients report their information in detail with enough time, but also made their visit physicians quickly grasp their conditions before a consultation []. It is apparent that the artificial intelligence-assisted conversational agents play a positive role in improving the efficiency and quality of health care delivery related to physicians.

Despite the application of artificial intelligence-assisted conversational agents in the tertiary public hospitals in economically developed regions of China, evidence to assess their effectiveness in improving health care delivery is lacking to date. Some studies investigated the factors influencing the adoption and continuance intention of patients toward artificial intelligence-assisted conversational agents in outpatient departments []. Several studies explored the design of the intelligence-assisted conversational agents [] and what issues and barriers exist in their usage []. Another study assessed the impact of artificial intelligence-based conversational agent on the operational performance []. However, there has been little further research that evaluates the effect of artificial intelligence-assisted conversational agents on the patient experience related to physicians during outpatient visits.

Therefore, this study aimed to examine whether the use of artificial intelligence-assisted conversational agents during outpatient visits improves the patient experience related to physicians and to further find out the difference in the patient experience between conversational agent users and nonusers.

Methods

Questionnaire Design

The Chinese Outpatient Experience Questionnaire was the basis of our survey, including 6 dimensions (physical environment and convenience, medical service fees, physician-patient communication, health information, short-term outcome, and general satisfaction), 28 items and participant characteristics (eg, sex, age, marital status, education, living place, monthly income, self-rated health status, and visit information) []. This outpatient experience questionnaire was verified with good reliability and validity (χ²/df =2.775, goodness-of-fit index=0.893, comparative fit index=0.930, Tucker-Lewis index=0.921, root mean square error of approximation=0.055, root mean square residual=0.038) []. However, we selected the above 4 dimensions (physician-patient communication, health information, short-term outcome, and general satisfaction; Cronbach alpha=.968 in this study) and the corresponding 19 items to survey the outpatient experience related to physicians. Moreover, we also added another question in the section of participant characteristics—“Did you use the artificial intelligence-assisted conversational agents during this outpatient visit?”—to divide the conversational agent users and nonusers.

Data Collection

The target population was adult residents who sought outpatient services from tertiary public hospitals within the past 2 weeks in China, selected using the random sampling. We used a professional data collection platform (Credamo) in China to create an electronic questionnaire in which to survey the targeted residents. The sample database of the Credamo included more than 3.0 million members with confirmed personal information from all provinces and regions in China []. With the support of Credamo, this study distributed electronic questionnaires to the targeted population nationwide and invited them to participate in the survey from April 1 to 15, 2025. Specifically, this study randomly sent the questionnaire links to the members who met the inclusion criteria nationwide through the Credamo in a targeted manner. These criteria were mainly set using the sample feature screening function of the Credamo as follows: being aged 18 years or older, being located within China, and having an outpatient experience in a tertiary public hospital within the past 2 weeks. Each invited participant could click on the link via their mobile phones to access and complete the electronic questionnaire. Before the survey, we introduced the nature and objective of the study and guaranteed that the collected data would not be used for other purposes. The survey was conducted accordingly after an individual’s consent was obtained. Each invited participant was prompted to fill in the electronic questionnaire based on their outpatient experience in tertiary public hospitals within the past 2 weeks. Each internet protocol address could be set to fill in the questionnaire only once.

Ethical Considerations

The institutional review board of Xuzhou Medical University approved this study before data collection (number 2024Z048). The general information about the nature and objective of this study was also provided and informed at the beginning of the survey as a means of informed consent. All the participants were informed that their participation was voluntary and they were free to refuse or discontinue their participation at any time. And only after an individual’s consent was obtained online could she or he continue to participate in this survey. During the data collection, no identifying information was collected, and the researchers only had access to the user ID. We also provided the participants who met the inclusion criteria and carefully completed the questionnaire with a monetary reward (US $0.70). This online survey was designed in accordance with the CHERRIES checklist.

Measures

The dependent variable was the total patient experience scores related to physicians in the multiple linear regression analysis. The four dimensions (physician-patient communication, health information, short-term outcome, and general satisfaction) and the corresponding 19 items of the Chinese Outpatient Experience Questionnaire were used to calculate the patient experience scores related to physicians during outpatient visits []. Each item was rated on a 5-point Likert scale, with a higher score indicating a better experience []. Each dimension score was calculated by adding up the scores of all items in the dimension and then dividing that sum by the total number of items in that dimension. We further calculated the total patient experience scores by summing the scores of the 19 items in the questionnaire and then dividing that sum by the total number of items []. Therefore, the total patient experience scores related to physicians ranged from 1 to 5. The independent variables included whether the artificial intelligence-assisted conversational agents were used during this outpatient visit (coded as 1=yes, 0=no), as well as the participant characteristics, including demographic and visit information.

Statistical Analysis

Descriptive statistics were performed to summarize data on the characteristics of participants. The t tests were then used to test the mean difference in the patient experience scores between conversational agent users and nonusers when the data followed a normal distribution. And multiple linear regression analysis was further performed to determine whether the use of artificial intelligence-assisted conversational agents during outpatient visits was associated with a better patient experience related to physicians. Moreover, the average percentage change in the dependent variable associated with a one-unit increase in an independent variable was calculated by dividing the independent variable’s unstandardized regression coefficient value by the mean value of the dependent variable and then multiplying by 100%. Benjamini-Hochberg adjusted P values ≤.05 were considered statistically significant. All data analyses were done using SPSS (version 23.0; IBM Corp) and STATA (version 15.0).

Results

Participant Characteristics of Conversational Agent Users and Nonusers

A total of 462 online responses were received, and 394 eligible responses remained, whereas 68 responses were excluded because they showed a certain logical contradiction based on the screening question (ie, whether you had an outpatient experience in a tertiary public hospital within the past 2 wk), or they contained the same answers to all questions, or because the time they were filled in was less than 120 seconds. The detailed characteristics of the participants are shown in . Among these participants, 53.0% (209/394) reported they used conversational agents during this outpatient visit. And the conversational agent users and nonusers differed in the sex (χ²₁=9.90, P=.002), educational level ( χ²₂=6.025, P=.049), monthly income (χ²₃=24.262, P<.001), self-rated health status ( χ²₂=31.247, P<.001) and physician title (χ²₃=9.643, P=.02). Moreover, these participants who rated their health status better were more likely to use the conversational agents during this outpatient visit.

Differences in Patient Experience Related to Physicians Between Conversational Agent Users and Nonusers

shows the patient experience scores of conversational agent users and nonusers. And the analysis results further showed that there was a significant difference in the total patient experience scores, the 4 dimensions, and the 19 items between the 2 groups.

Specifically, in the total patient experience scores related to physicians, conversational agent users obtained significantly higher scores than nonusers (t₃₉₂=5.589, P<.001). In these dimensions of physician-patient communication (t₃₉₂=5.013, P=.006), health information (t₃₉₂=5.758, P<.001), short-term outcome (t₃₉₂=4.608, P<.001) and general satisfaction (t₃₉₂=5.080, P<.001), conversational agent users reported significantly higher scores than nonusers as well.

Moreover, conversational agent users also reported significantly higher scores than nonusers in the 19 items of patient experience related to physicians (see ).

Influence of Conversational Agents on Patient Experience Related to Physicians

As shown in , after controlling for other factors on participant characteristics including demographic and visit information and adjusting the P value using Benjamini-Hochberg procedure, whether the conversational agent was used or not during this outpatient visit was a significant factor influencing the total patient experience scores related to physicians (B=0.298, P=.013). The standardized regression coefficient of whether the conversational agent was used was 0.205. Thus, when other covariates were held constant, the use of the artificial intelligence-assisted conversational agents averagely increased the total patient experience scores related to physicians by 7.51% (0.298/3.97*100%).

Among these control factors, self-rated health status (B=0.520, P=.013) was a significant factor that influenced the total patient experience scores related to physicians. And these residents who rated their health status as “very good” were more likely to report a higher patient experience score related to physicians during this outpatient visit.

Moreover, the regression model explained 25.54% of the variance in the total patient experience scores related to physicians (R²=0.2554). And we further calculated the value of the variance inflation factor to check for collinearity. The variance inflation factor value of all independent variables was between 1.15 and 2.51, which indicated that there was no collinearity.

Discussion

Principal Findings

We found that the use of artificial intelligence-assisted conversational agents averagely increased the total patient experience scores related to physicians by 7.51%. In this study, conversational agent users reported a better experience in the physician-patient communication, access to health information, short-term outcomes, and general satisfaction as well as their specific 19 items.

The use of artificial intelligence-assisted conversational agents can improve communication efficiency between physicians and patients during outpatient visits. After completing the registration, the patients can click on the “pre-consultation” on the registration and appointment page of the hospital’s mobile app by using their mobile phones and then engage in a dialog with artificial intelligence chatbots []. And a corresponding structured preconsultation report is formed and delivered to their visit physicians, and the physicians thereby quickly grasp the patients’ conditions before a consultation and further conduct a targeted inquiry []. This could improve the consultation efficiency between physicians and patients and, in turn, contribute to positive outcomes, such as making physician-patient communication better, accessing more targeted health information, ameliorating short-term outcomes, and increasing general satisfaction. These positive outcomes that appear to result from the use of the artificial intelligence-assisted conversational agents have been confirmed in our quantitative study.

Currently, the situation where Chinese patients have a lack of adequate communication with their physicians during outpatient visits in the tertiary hospitals has not been effectively improved, which hinders the improvement of the current physician-patient relationship [,]. The artificial intelligence-assisted conversational agents can help patients communicate more effectively with their visit physicians and access more targeted health information within the existing limited time. This could, in turn, result in a better physician-patient relationship during outpatient visits.

Our quantitative study found that the overall patient experience related to physicians could be improved significantly when the artificial intelligence-assisted conversational agent was used during outpatient visits. This finding could be supported by the findings of Lu et al [] regarding the effect of mobile health apps on the patient experience that using mobile health apps could improve the patient experience. And the extent to which the artificial intelligence-assisted conversational agents improved the patient experience in this study was higher than that of the mobile health apps reported in the previous research in 2018 (7.51% vs 5.35%) []. This difference might be relevant to the fact that the artificial intelligence-assisted conversational agents not only allow the patients to report their information in detail before a consultation, but also further make them communicate more efficiently with their visit physicians within the existing limited time [,], thereby bringing them a better communication experience with their visit physicians. In contrast, the past mobile health apps were dedicated to saving the patients’ waiting time throughout their visits and thereby improving their visit experience []. More importantly, there is increasing evidence supporting that improved health care system delivery could improve the patient experience, which in turn brings a better health outcome to patients [,]. Therefore, we have reason to believe that the increased application and use of the artificial intelligence-assisted conversational agents in outpatient departments could contribute to a better health outcome for outpatients.

Nevertheless, the current application of artificial intelligence-assisted conversational agents in outpatient departments is mainly in the tertiary public hospitals in big cities of China (eg, Shanghai, Shenzhen, and Wuhan). And the existing survey research in 2022 found that six months after tertiary hospitals in Shanghai deployed the artificial intelligence-assisted conversational agents in outpatient departments, the patients’ usage rates fell short of expectations (26% and 20% for the second- and fourth-ranked hospitals, respectively) []. Therefore, we suggest that public hospitals should be encouraged to promote the application of the artificial intelligence-assisted conversational agents in outpatient departments and integrate them into the functions of their existing mobile health apps to continuously improve the patient experience related to physicians during outpatient visits. More importantly, given that public hospitals in less-developed regions generally lack sufficient funds to deploy the artificial intelligence-assisted conversational agents, we also suggest that the Chinese government should increase financial support for these public hospitals. This would accelerate the promotion of the artificial intelligence-assisted conversational agents in outpatient departments and thereby improve the patient experience on a large scale.

Moreover, our study also showed that self-rated health status was a significant factor influencing the patient experience related to physicians during outpatient visits. This result is similar to the findings of several studies on the effect of mobile health apps on the patient experience that the patients who rated their health status better were more likely to report a better patient experience []. Another study by Li et al also indicated that the patients with worse self-rated health status would be more likely to experience a negative physician-patient relationship []. Therefore, we also suggest that hospitals should make full use of the artificial intelligence-assisted conversational agents to further improve the medical experience of these patients with worse self-rated health status.

Limitations

There are some limitations in this study. First, data collection was self-reported by adult residents based on their outpatient experience within the past 2 weeks, which might have a recall bias and selection bias. Second, our conclusions might have been biased by distributions, such as sex and age. Therefore, after controlling for the influence of the confounding factors on participant characteristics, multiple linear regression analysis was performed to examine whether the use of artificial intelligence-assisted conversational agents improves the patient experience related to physicians during outpatient visits, which could have resulted in a reliable and stable conclusion. Third, patients can freely choose to use or not use the artificial intelligence-assisted conversational agents during outpatient visits, which might be influenced by these factors (eg, digital literacy, education, and general attitude toward health technology). And this might also act as a confounder in masking the patient experience. Furthermore, further research is necessary to explore the intrinsic mechanism by which the use of artificial intelligence-assisted conversational agents improves the patient experience related to physicians during outpatient visits.

Conclusions

Our work provides evidence supporting the use of artificial intelligence-assisted conversational agents for improving the patient experience related to physicians during outpatient visits, especially in terms of making physician-patient communication better, accessing more targeted health information, ameliorating short-term outcomes, and increasing general satisfaction. All of these may further bring positive health outcomes to patients. Therefore, we suggest that public hospitals should consider the benefits of the artificial intelligence-assisted conversational agents and actively deploy the conversational agents in outpatient departments so as to continuously improve the patient experience related to physicians during outpatient visits.