This study was conducted in three phases, in accordance with the Design and Development Research (DDR) framework [16]. Phase 1 (not described in this article) involved a need analysis via literature review to identify gaps in students’ learning interest, understanding, and engagement of the occupational health subject as well as the need of a novel programme. Phase 2 involved the design and development of a metaverse programme known as MATOSH. In Phase 3, the MATOSH programme was implemented among medical students to empirically evaluate its effectiveness in improving their interest, understanding, and engagement in the occupational health subject. The study flowchart is illustrated in Fig. 1 below.
Flowchart of the Current Study
Phase 2 (Design and Development)
Phase 2 involved two components, which were the design and development of the MATOSH programme. This phase was conducted over a two-month period from June 2024 to July 2024.
Study design, study location, and study population
The content design of the MATOSH programme adopted a nominal group technique (NGT). Third-year medical students from the Faculty of Medicine at the National University of Malaysia served as the study population. This cohort was selected because the occupational health subject was introduced to them for the first time during the public health posting. Meanwhile, the development of MATOSH involved both academicians and third-year medical students from the same university, all of whom participated exclusively through online meetings.
Sample size
Fifteen third-year medical students and an occupational health lecturer from the Faculty of Medicine at the National University of Malaysia (UKM) were invited to the NGT to identify the top five important occupational health topics to be included in the MATOSH programme. The development of MATOSH involved three academicians (i.e., one occupational health and two computer science lecturers) and three third-year medical students.
Sampling method
The 15 medical students in NGT were recruited through purposive sampling. The sampling frame, consisting of a list of medical students, was obtained from the posting coordinator for the public health posting. Students (the sampling units) were selected based on the following inclusion criteria: (i) completion of the public health posting; (ii) attendance of at least 75% of the occupational health lectures (i.e., three out of four lectures); and (iii) access to a metaverse platform, regardless of prior experience with its use. However, students were excluded if they: (i) had recently failed the public health posting; or (ii) declined to provide informed consent for participation in the study. An experienced occupational health lecturer who currently teaching the subject at the same university was also invited to the NGT. For the development of MATOSH, the developer team (consisted of three academicians and three medical students) were recruited through purposive sampling method from the researchers’ network.
Study instruments
To determine and include the top five important occupational health topics for the content design of the MATOSH programme, an interview schedule was developed. It consisted of a central question of “what topics should be included in the MATOSH programme to effectively teach occupational health to medical students”. Meanwhile, to develop the MATOSH programme, a password secured laptop (Microsoft Surface Go with 16GB RAM) was utilised. Besides that, online software such as Spatial.io and Sketfab.com were used to design the virtual environment and virtual avatars, respectively. Both of these software were used under free account plans.
Study procedure
For the NGT (for MATOSH programme content design), 15 third-year medical students who met the inclusion criteria and an occupational health lecturer were invited to a meeting room at the faculty. As an introduction, the participants were brief regarding the aim of the NGT, which was to determine the top five important occupational health topics to be included in the MATOSH programme. Initially, participants engaged in silent idea generation, where they independently wrote down as many relevant occupational health topics as they could within 10 min. This was followed by a round-robin sharing session, where each participant took turns presenting one idea at a time. All ideas were recorded verbatim by the facilitator on a shared screen. This continued until all participants had contributed all their ideas. No discussion or evaluation occurred during this stage to ensure equal contribution from all participants. In the clarification and consolidation phase, the participants reviewed the compiled list of occupational health topics. Any overlapping or similar ideas were grouped or rephrased with the consensus of the participants. Participants could ask questions for clarification, but detailed debates were discouraged to maintain focus and efficiency. Next, during the voting and ranking phase, each participant was asked to provide scores for each occupational health topic based on importance (“1” = least important until “5” = most important). Rankings were submitted anonymously using Google form. In the final step, results were tallied, and the five topics with the highest percentages were identified and displayed. A group discussion followed to confirm consensus and ensure all participants were satisfied with the outcome. This process resulted in a ranked list of the top five important occupational health topics, which served as the foundation for content design of the MATOSH programme.
Once the top five important occupational health topics were identified, the development of a metaverse programme began by following a structured seven-step process [17]. The first step involved defining the metaverse concept by aligning content with the existing occupational health subject, which included four lectures on occupational hazards, HIRARC, SOCSO roles, relevant legislation, and universal precautions in hospital. Content development was guided by consultations with one occupational health lecturer and two computer science lecturers. The second step established the technological requirements, which identify the necessary hardware and software components. MATOSH was developed using Spatial.io [18], a free web-based platform that supported metaverse creation without the need for advanced devices such as VR or AR goggles, and without requiring personal cloud storage, as all data resided on the platform itself. The third step focused on designing virtual entities and environments. The virtual space was modelled as a hospital using Spatial.io templates, while virtual three dimensional hospital staff were generated from Sketchfab.com [19]. The fourth step integrated social interaction dynamics to enhance student engagement. As such, students were able to customise avatars, navigate through the virtual hospital using “W””, “A”, “S”, “D” buttons, interact with other users, and dicuss occupational health scenarios virtually. The fifth step addressed the development of a virtual economy, in which MATOSH was decided as a non-profit educational tool with no commercial or NFT elements. The sixth step involved testing and optimisation through internal evaluation and feedback from three medical students, which allowed for improvements in usability and content. The final step was the launch and publication of MATOSH on Spatial.io, with access restricted to educational and research use, and no online promotional efforts.
Statistical analysis
In the NGT, the total scores for each occupational health topic were calculated and converted to percentages (i.e., total score of each topic divided by 75 and then convert to 100%), and the five occupational health topics with the highest percentages were selected. Descriptive statistics, including frequency counts and total scores, were used to summarise the ranking data. No inferential statistical tests were applied, as the purpose of the NGT was to reach group consensus rather than to test hypotheses.
Phase 3 (Implementation and Evaluation)
Phase 3 consisted of two components, namely the implementation and evaluation of the effectiveness of MATOSH in improving medical students’ interest, understanding, and engagement in the occupational health subject. This phase was conducted over a 9-month period, from August 2024 to April 2025.
Study design, study location, and study population
The implementation component of this phase of the study involved a cognitive debriefing to validate the MATOSH programme, while the evaluation component of the MATOSH programme employed a quasi-experimental design. Third-year medical students from the Faculty of Medicine at the National University of Malaysia served as the study population. This cohort was selected because they were introduced to the occupational health subject for the first time during the public health posting.
Sample size Estimation
Given the skill engagement score for students with conventional learning = 3.78 ± 0.55 points [20], skill engagement score for students with e-learning = 3.56 ± 0.81 points [20], significance level = 0.05, power of study = 80%, the required sample size was 208 students (i.e., 104 students for intervention and 104 students for control group). This was calculated using Pocock’s formula for two means. Although 208 students were recruited, 36 did not return their responses, resulted only 88 and 84 students in the intervention and control groups, respectively.
Sampling method
In Phase 3, medical students were recruited through purposive sampling. The sampling frame, which was the list of medical students, was obtained from the posting coordinator for the public health posting. Students (the sampling units) were selected based on the following inclusion criteria: (i) enrolment in the public health posting at the time of recruitment; (ii) attendance of at least 75% of the occupational health lectures (i.e., three out of four lectures); and (iii) access to a metaverse platform, regardless of prior experience with its use. Exclusion criteria included: (i) students who were repeating the public health posting at the time of recruitment; and (ii) students who declined to provide informed consent for participation in the study.
Study instruments
In the implementation component of Phase 3, a semi-structured interview schedule was developed for the cognitive debriefing. This interview schedule included six questions pertaining to the content clarity, content relevance, visual appearance, usability, education value, technical functionality.
Meanwhile, in the evaluation component of Phase 3, three tools were utilised to evaluate the effectiveness of the MATOSH: the Study Interest Questionnaire (SIQ), Occupational Health End of Module (OH-EOM) test paper, and the Student Course Engagement Questionnaire (SCEQ). These instruments were employed to evaluate students’ interest, understanding, and engagement in the occupational health subject, respectively.
The Study Interest Questionnaire (SIQ) is an 18-item, unidimensional questionnaire developed by Schiefele et al. in 1987 [21]. It was intended to measure students’ interest in a specific field of study, based on the Educational Interest Theory. The 18 items in the SIQ were ranked on a Likert scale from 1 (“not at all true”) to 4 (“completely true”), with scores ranging from 18 to 72 points. Higher total scores were associated with more interest in the student’s field of study. Items 1, 2, 5, 7, 9, 10, and 11 were reverse coded (see Appendix C). Cronbach’s alpha was 0.90 and test-retest reliability was 0.67 over a 2-year period. Convergent, discriminant, and concurrent validity of the SIQ was demonstrated by correlations between SIQ and intrinsic motivation (r = 0.46, p < 0.001), extraversion (r = 0.01, p > 0.05), and use of deep learning strategies (r = 0.45, p < 0.001). Students’ exam performance over 2 years correlated with SIQ scores as evidence of predictive validity (r = 0.33, p < 0.05).
The Occupational Health End of Module (OH-EOM) test paper was created by the lecturers responsible for teaching the occupational health subject in the public health posting. This test was initially developed to assess students’ understanding of the occupational health subject at its conclusion. It comprised 10 multiple-choice questions (with options A to D), and students were required to select the best answer. Each correct response was awarded one mark, resulting in a total score range of 0 to 10. Higher marks indicated a higher level of understanding towards the occupational health subject. This test paper was vetted by two occupational health lecturers at the Department of Public Health Medicine, Faculty of Medicine, UKM.
The Student Course Engagement Questionnaire (SCEQ) was developed by Handelsman in 2005 to assess engagement among university students [22]. It consisted of 23 items with four domains, including skills engagement (9 items), participation/interaction engagement (6 items), emotional engagement (5 items), and performance engagement (3 items) (Handelsman et al. 2005). Participants responded using a 5-point Likert scale (i.e., 1 = “much less like me” to 5 = “much more like me”) (see Appendix E). The total scores on the SCEQ ranged from 23 to 115. Although there was no specific cut-off point in this questionnaire, higher total scores indicated a higher level of engagement. The SCEQ demonstrated good internal consistency across the four engagement domains (Cronbach’s alpha ranged from 0.76 to 0.82). It also demonstrated good convergent validity, in which the SCEQ scores were significantly associated with absolute and relative engagements (β = 0.16–0.38).
Study procedure
For the implementation component of Phase 3, 15 third-year medical students who met the inclusion criteria (similar to those described in Phase 2) were invited to a lecture hall for cognitive debriefing. They were first requested to independently explore the MATOSH programme using their personal devices (e.g., smartphones, tablets or laptops). Following this, each student participated in a cognitive debriefing interview. The feedback obtained from these interviews informed essential modifications to enhance the design, functionality, and usability of the metaverse. All cognitive debriefing interviews were audio-recorded to serve as a reference for the research team.
For the evaluation component of Phase 3, medical students who were assigned to the control group (n = 104) received only conventional lectures. Meanwhile, the medical students allocated to the intervention group (n = 104) received both conventional lectures and the MATOSH programme.
For the control group, four occupational health lectures were conducted, with one lecture per week in a lecture hall. The four lectures covered the following topics: types of occupational hazards, HIRARC, the roles of SOCSO, occupational safety and health legislations, and universal precautions in hospital. At the conclusion of the posting, the researchers distributed the study instruments (SIQ, OH-EOM test paper, and SCEQ) for in-class assessment. Prior to participation, students were briefed on the research project and asked to complete an online informed consent form. Those who declined participation were instructed to remain seated quietly in the lecture hall. Medical students who consented to participate received three Google Forms, namely the SIQ, OH-EOM, and SCEQ. They were instructed to complete all tasks within 20 min without engaging in peer discussion.
Medical students in the intervention group also attended four weekly occupational health lectures, covering the same topics as the control group: types of occupational hazards, HIRARC, the roles of SOCSO, occupational safety and health legislations, and universal precautions in hospital. Additionally, the intervention group had an extra session dedicated to introducing them to the MATOSH metaverse programme. During this fifth session, students received instructions on how to navigate the MATOSH platform and were given one hour to explore its content using their own mobile devices (e.g., smartphones, tablets or laptop). At the end of the posting, the researcher distributed the study instruments (SIQ, OH-EOM test paper, and SCEQ) for in-class assessment. Students were briefed on the research project and asked to complete an online informed consent form before participating. Those who chose not to participate were instructed to remain quietly in the lecture hall. Students who consented to participate received three Google Forms, namely the SIQ, OH-EOM, and SCEQ, and were asked to complete all tasks within 20 min, without engaging in peer discussion.
Statistical analysis
Qualitative data from the implementation component (cognitive debriefing) was analysed thematically using inductive coding. Meanwhile, in the evaluation component, descriptive analysis was conducted to outline the sociodemographic characteristics of the medical students who participated in the study. Continuous data were reported as mean and standard deviation if data were normally distributed, or as median and interquartile range if data were not normally distributed. Categorical data were presented as frequencies and percentages.
Since all data was normally distributed (ascertained using Kolmogorov-Smirnov test), a paired t-test was conducted to assess any significant difference between pre- and post-intervention SIQ, OH-EOM, and SCEQ scores for both intervention and control groups (i.e., within group comparison). Meanwhile, an independent t-test was performed to assess the significant differences in the SIQ, OH-EOM, and SCEQ scores between the intervention and control groups (i.e., between groups comparison). A significance level of p < 0.05 was set. All statistical analyses were conducted using SPSS version 28.
Ethical considerations
The current study received ethical approval from the Ethics Committee of the National University of Malaysia (JEP-2025-219). Online information booklets outlining the study’s objectives and procedures were provided to the medical students via Google Forms. Online informed consent was obtained from the medical students prior to their participation. Medical students were not penalised if they refused to participate in the study. They were also taught with the same level of commitment and the same amount of academic content, regardless of their group allocation. The investigators adhered to the principles outlined in the Declaration of Helsinki as well as the Malaysian Good Clinical Practice Guidelines.
No personal identifiers were captured in the Google Form. Only authorised research team members had access to the research data, and their access was monitored and restricted based on necessity. The collected data were securely stored in a password-protected, encrypted online database hosted on a secured server. Study data were retained for a period of five years following the publication of the study results. At the end of the five-year retention period, all study data were permanently deleted or destroyed.