NONE 20260630062817169 naun/naun content-registration-form@crossref.org content-registration-form International Journal of Education and Information Technologies 2074-1316 Andrei Stefan Iacob National University of Science and Technology POLITEHNICA Bucharest, Faculty of Industrial Engineering and Robotics, 313 Splaiul Independenței, Sector 6, Zip Code 060042, Bucharest, Romania Oana-Roxana Chivu National University of Science and Technology POLITEHNICA Bucharest, Faculty of Industrial Engineering and Robotics, 313 Splaiul Independenței, Sector 6, Zip Code 060042, Bucharest, Romania Mihnea Costoiu National University of Science and Technology POLITEHNICA Bucharest, Faculty of Industrial Engineering and Robotics, 313 Splaiul Independenței, Sector 6, Zip Code 060042, Bucharest, Romania Alina Trifu National Research and Development Institute on Occupational Safety - I.N.C.D.P.M. “Alexandru Darabont”, 35A Ghencea Blvd., Zip Code 061692, Bucharest, Romania Roland Moraru University of Petroșani, Faculty of Mining, Department of Management and Industrial Engineering, Petrosani, Zip Code 332006, Hunedoara, Romania Adrian Bibiș National Research and Development Institute for Gas Turbines COMOTI, B-dul Iuliu Maniu nr. 220D, sector 6, Zip Code 061126, Bucharest, Romania Digital occupational safety and health training is nowadays widely adopted in complex socio-technical systems. Yet its effectiveness is often assessed using linear approaches that fail to capture impacts on real work and system performance. This study argues that training effectiveness cannot be meaningfully evaluated without a systemic perspective. To address this gap, the paper proposes the FRAM-Kirkpatrick Framework, integrating the Functional Resonance Analysis Method (FRAM) with the Kirkpatrick training evaluation model. The framework conceptualizes digital OSH training as a systemic intervention influencing functional performance variability. The FRAM-Kirkpatrick Framework is applied through multiple case studies in the road transport sector involving three companies and 47 professional drivers. Results show significant improvements in learning and safe behavior, accompanied by a substantial reduction in functional variability within critical system functions. The findings demonstrate that the FRAM-Kirkpatrick Framework enables a more robust evaluation of digital training effectiveness in complex socio-technical systems. 06 30 2026 06 30 2026 124 13 https://creativecommons.org/licenses/by/4.0/deed.en_US 10.46300/9109.2026.20.13 https://npublications.com/journals/educationinformation/2026/a262008-013(2026).pdf C. Büscher (2022). “The Problem of Observing Sociotechnical Entities in Social Science and Humanities Energy Transition Research”, Frontiers in Sociology, volume 6, pp. 699362, doi: https://doi.org/10.3389/fsoc.2021.699362 I. T. de Souza, A. C. Rosa, M. C. R. Vidal, M. K. Najjar, A. W. A. Hammad, (2021). “Information Technologies in Complex Socio-Technical Systems Based on Functional Variability: A Case Study on HVAC Maintenance Work Orders”. Applied Sciences, volume 11, issue 3, pp. 1049, 2021, doi: https://doi.org/10.3390/app11031049 P. M. Salmon, Gemma J.M. Read, Nicholas J. Steven, “Who is in control of road safety? A STAMP control structure analysis of the road transport system” Accident Analysis & Prevention, volume 96, pp. 140-151, 2016, doi: https://doi.org/10.1016/j.aap.2016.05.025 G.J. Casey, T. Miles-Johnson, G.J. Stevens “I'm Not Right to Drive, but I Drove out the Gate": Personal and Contextual Factors Affecting Truck Driver Fatigue Compliance”, International Journal of Environmental Research and Public Health, volume 22, issue 11, pp. 1724, 2025, doi: https://doi.org/10.3390/ijerph22111724 L. N. Langendorf, S. Khalid, “Systematic literature review on usability and training outcomes of using digital training technologies in industry”, Computers in Human Behavior Reports, volume 17, pp. 100604, 2025, doi: https://doi.org/10.1016/j.chbr.2025.100604 T. H. Dao, A. C. Sisavath, , H. T. T Bui, “Digital Workforce Training, Employee Motivation, Job Satisfaction, Digital Behavior as Determinants of Employee Performance”, Sage Open, volume 15, issue 4, 2025. doi: https://doi.org/10.1177/21582440251395698 A. Badriya, Fung, Chorng Yuan, Turner, K., Lim, Agnes, “A Critical Review on Training Evaluation Models: A Search for Future Agenda”, Journal of Cognitive Sciences and Human Development, volume 10, issue 1, pp. 142- 170, 2024, doi: https://doi.org/10.33736/jcshd.6336.2024 H. Stefan, M. Mortimer, B. Horan, “Evaluating the effectiveness of virtual reality for safety-relevant training: a systematic review”. Virtual Reality, volume 27, pp. 2839–2869, 2023, doi: https://doi.org/10.1007/s10055- 023-00843-7 H. Urbancova, H. Vrabcová, “Effective Training Evaluation: The Role of Factors Influencing the Evaluation of Effectiveness of Employee Training and Development”, Sustainability, volume 13, issue 5, pp. 2721, 2021, doi: https://doi.org/10.3390/su13052721. P. Carayon, P. Hancock, N. Leveson, I. Noy, L. Sznelwar, G. van Hootegem, “Advancing a sociotechnical systems approach to workplace safety – developing the conceptual framework”, Ergonomics, volume 58, issue 4, pp. 548– 564, 2015, 548–564, Accessed Date: 03, 22, 2025, doi: http://dx.doi.org/10.1080/00140139.2015.1015623. A. Bayramova, D. J. Edwards, C. Roberts, I. Rillie, “Enhanced safety in complex socio-technical systems via safety-in-cohesion”, Safety Science, volume 164, pp. 106176 2023, doi: https://doi.org/10.1016/j.ssci.2023.106176 R. Patriarca, G. Di Gravio, R. Woltjer, F. Costantino, G. Praetorius, P. Ferreira, E. Hollnagel, “Framing the FRAM: A literature review on the functional resonance analysis method”, Safety Science, volume 129, pp. 104827, 2020, doi: https://doi.org/10.1016/j.ssci.2020.104827. E. Hollnagel, “Prologue: The Scope of Resilience Engineering” Resilience Engineering in Practice: A Guidebook, Ashgate Publishing Company, Bulington, ISBN 978-1409410355, 2011, https://api.pageplace.de/preview/DT0400.9781317065265 _A34254892/preview-9781317065265_A34254892.pdf. [Access Date: 01/01/2026] A. Aljawharah, C. Callinan, “Adaptation of Kirkpatrick’s Four-Level Model of Training Criteria to Evaluate Training Programmes for Head Teachers” Education Sciences volume 11, issue 3, pp. 116, 2021, doi: https://doi.org/10.3390/educsci11030116 M. Cahapay, “Kirkpatrick Model: Its Limitations as Used in Higher Education Evaluation”, International Journal of Assessment Tools in Education, volume 8, issue 1, pp. 135-144, 2021, doi: https://doi.org/10.21449/ijate.856143 A. Cordeiro, Y. Ferreira, R. Leite, L. Almeida, M. Catapan, A. Siqueira, T. Silva “Immersive technologies for evaluating industrial safety training in high-risk environments: a review on opportunities and challenges”. Frontiers. Virtual Reality, volume 6, pp. 1726058, 2025, doi: https://doi.org/10.3389/frvir.2025.1726058 J.M. Spector, “Conceptualizing the emerging field of smart learning environments”, Smart Learning Environment, volume 1, issue 2, pp. 2, 2014, doi: https://doi.org/10.1186/s40561-014-0002-7. A. Babalola, P. Manu, C. Cheung, A. Yunusa-Kaltungo, P. Bartolo, “Applications of immersive technologies for occupational safety and health training and education: A systematic review”, Safety Science, volume 166, pp. 106214, 2023, doi: https://doi.org/10.1016/j.ssci.2023.106214. D. Scorgie, Z. Feng, D. Paes, F. Parisi, T.W. Yiu, R. Lovreglio, “Virtual reality for safety training: A systematic literature review and meta-analysis”, Safety Science, volume 171, pp. 106372, 2024, doi: https://doi.org/10.1016/j.ssci.2023.106372. D. E. Itodo, S. O. Oshodi, N. I. Nwankwo, F. A. Emuze, E. Chinyio, “The Use of Digital Technologies in Construction Safety: A Systematic Review” Buildings, volume 15, issue 8, pp. 1386, 2025, doi: https://doi.org/10.3390/buildings15081386 A. M. Vukićević, I. Mačužić, M. Djapan, , Milićević, V., L. Shamina, “Digital Training and Advanced Learning in Occupational Safety and Health Based on Modern and Affordable Technologies”, Sustainability, volume 13, issue 24, pp. 13641, 2021, doi: https://doi.org/10.3390/su132413641 R. Rampun, Z. Zainol, D. Tajuddinc, “The effects of training transfer on training program evaluation and effectiveness of training program. Management Research Journal”, volume 9, pp. 43–53, 2020, doi: https://doi.org/10.37134/mrj.vol9.sp.4.2020 H. Yaseen, A. S. Mohammad, N. Ashal, H. Abusaimeh, “The Impact of Adaptive Learning Technologies, Personalized Feedback, and Interactive AI Tools on Student Engagement: The Moderating Role of Digital Literacy”. Sustainability, volume 17, issue 3, pp. 1133, 2025, doi: https://doi.org/10.3390/su17031133. A. O'Neill, “Transfer of workplace e-learning: A systematic literature review”, Social Sciences and Humanities Open, volume 11, pp. 101407, 2025, doi: https://doi.org/10.1016/j.ssaho.2025.101407. K. H. Wang, “Sustainable E-learning? A flash in the pan or a lasting change for workers”, World Development Sustainability, volume 7, pp. 100258, 2025, doi: https://doi.org/10.1016/j.wds.2025.100258. S. Kauffeld, J. Decius, C. Graßmann, “Learning and Transfer in Organisations: How It Works and Can Be Supported.” European Journal of Work and Organizational Psychology, volume 34, issue 2, pp. 161– 74, 2025, doi: F. Fauth, J. González-Martínez, “The Work Environment and Its Influence on Learning Transfer in Virtual InService ICT Training for Teachers and University Professors” Education Sciences volume 13, issue 4, pp. 330, 2023, doi: https://doi.org/10.3390/educsci13040330. B.D. Blume, J. Kevin Ford, E.A. Surface, J. Olenick, „A dynamic model of training transfer, Human Resource Management Review”, Advancing Training for the 21st Century, volume 29, issue 2, 2019, pp. 270-283, doi: https://doi.org/10.1016/j.hrmr.2017.11.004. B. Trenerry, S. Chng, Y. Wang, Z. S. Suhaila, S.S. Lim, H.Y. Lu “Preparing Workplaces for Digital Transformation: An Integrative Review and Framework of Multi-Level Factors”. Frontiers. Psychology, volume 12, issue 1, pp. 620766, 2021, https://ink.library.smu.edu.sg/cis_research/84/, [Access Date: 01/01/2206]. E. Salas, S. I. Tannenbaum, K., Kraiger, K. A. SmithJentsch, “The science of training and development in organizations”. Psychological Science in the Public Interest, volume 13, issue 2, pp. 74–101, 2012, doi: https://doi.org/10.1177/1529100612436661. A. Grohmann, S. Kauffeld, “Evaluating training programs: Development and correlates of the Questionnaire for Professional Training Evaluation”. International Journal of Training and Development, volume 17(2), pp. 135– 155, 2013, https://psycnet.apa.org/record/2013-15765- 003, [Access Date: 01/01/2026]. T. Sitzmann, K. G. Brown, W. J. Casper, K., R. D. Zimmerman, “A review and meta-analysis of the nomological network of trainee reactions”. Journal of Applied Psychology, volume 93, issue 2, pp. 280–295, 2011, doi: https://doi.org/10.1037/0021-9010.93.2.280 B. D. Blume, J. K. Ford, T. T. Baldwin, J. L. Huang “Transfer of training: A meta-analytic review”. Journal of Management, volume 36, issue 4, pp. 1065–1105, 2010, doi: https://doi.org/10.1177/0149206309352880 R. Bates, “A critical analysis of evaluation practice: The Kirkpatrick model and the principle of beneficence”, Evaluation and Program Planning, volume 27, issue 3, pp. 341–347, 2004. doi: https://doi.org/10.1016/j.evalprogplan.2004.04.011 L. Praslova, “Adaptation of Kirkpatrick’s four level model of training criteria to assessment of learning outcomes and program evaluation in Higher Education.” Educational Assessment, Evaluation and Accountability, volume 22, pp. 215–225, 2010, doi: https://doi.org/10.1007/s11092-010-9098-7. E. Hollnagel, “FRAM: The Functional Resonance Analysis Method: Modelling Complex Socio-Technical Systems”, eBook ISBN 9781315255071, 2012, doi: https://doi.org/10.1201/9781315255071. E. Hollnagel, “Safety-I and Safety-II: The Past and Future of Safety Management.” Cognition Technology and Work, volume 17, pp. 461-464, 2015, doi: https://doi.org/10.1007/s10111-015-0345-z T. Bjerga, T. Aven, E. Zio, “ Uncertainty treatment in risk analysis of complex systems: The cases of STAMP and FRAM”, Reliability Engineering & System Safety, volume 156, pp. 203-209, 2016, doi: https://doi.org/10.1016/j.ress.2016.08.004. L. V. Rosa, A. N. Haddad, P. V. R. de Carvalho, “Assessing risk in sustainable construction using FRAM”, Cogn Tech Work, volume 17, pp. 559–573, 2015, doi: https://doi.org/10.1007/s10111-015-0337-z. R. Patriarca, G Di Gravio, F. Costantino, “A Monte Carlo evolution of the FRAM to assess performance variability”. Safety Science, volume 91, pp. 49–60, 2017, doi: https://doi.org/10.1016/j.ssci.2016.07.016.