A risk-based audit framework for performance-driven on-the-job training quality in computing education: Evidence from a three-year sequential cohort study

Brenda M. Balala & Eduardo R. Yu

Volume 7 Issue 3, September 2026

Abstract

In the Philippines, the quality of on-the-job training (OJT) in computing programs has generally been assessed using narrative supervisor ratings and completion checklists, instead of psychometrically validated instruments. This study conceptualizes the Committee of Sponsoring Organizations of the Treadway Commission’s (COSO) Enterprise Risk Management (ERM) framework, Industry 4.0 readiness, BADIR analytics, and Kolb’s Experiential Learning Theory (ELT) into a single conceptual framework to develop and validate an OJT assessment tool for governance purposes. A sequential-cohort, quantitative-dominant mixed-methods design with an embedded qualitative phase was used over a three-year period (SY 2023-2024 to 2025-2026), with a sample of 64 of 99 qualified trainees at a private Catholic educational institution in the Philippines. The instrument comprised eight constructs measured through five-point Likert-scale items and four open-ended items, analyzed using reliability, descriptive, comparative, correlational, and thematic analyses. The instrument demonstrated high overall reliability (Cronbach’s alpha = .928), with construct means ranging from 4.30 to 4.62 and an Overall OJT Quality mean of 4.57. Analysis of variance confirmed temporal stability, showing no significant differences across cohorts. Key Performance Indicator (KPI)-Driven Performance and Control Environment showed the strongest correlations with Overall OJT Quality, and thematic analysis of open-ended responses largely confirmed these quantitative trends. The BSCS-specific Industry 4.0 subscale showed weak internal consistency and requires further refinement. Findings are exploratory, given the single-institution, modest sample, warranting larger multi-institutional validation. The results suggest that enterprise risk-management principles can be feasibly adapted into a psychometrically supported governance tool for OJT quality assurance in Philippine computing education.

Keywords

Author information & Contribution

Brenda M. Balala. Corresponding author. Master in Information Technology (MIT), Associate Professor, Notre Dame of Marbel University; Doctor of Information Technology (DIT) candidate, Graduate School, AMA University. Email: bmbalala@ndmu.edu.ph

Eduardo R. Yu. Doctor of Information Technology (DIT), Associate Professor, Graduate School, AMA University; Doctoral Research Adviser to the first author under the Research-to-Publication Integration Strategy (RPIS). Email: Eduardo.yu@ama.edu.ph

Brenda M. Balala (B.M.B.), as DIT student and principal investigator, was responsible for acquisition, analysis, and interpretation of data and led the drafting of the work: she framed the research problem and finalized the study title (RPIS Week 1); conducted the literature synthesis and constructed the conceptual framework (Week 2); designed the methodology and developed and validated the KPI-Driven OJT Evaluation Instrument (Weeks 3–4); administered data collection across three sequential cohorts and prepared the cleaned dataset (Weeks 5–6); performed the statistical and thematic analyses and generated the results (Week 7); interpreted the findings against the reviewed literature (Week 8); and drafted the complete IMRAD manuscript, conducted the similarity and citation checks, and prepared and submitted the manuscript for publication (Weeks 9–11).
Eduardo R. Yu II (E.R.Y.II), as DIT-holder and doctoral research adviser, revised the work critically for important intellectual content by performing the Faculty Validation function specified at each RPIS milestone: he evaluated the feasibility and gap-alignment of the proposed research titles and problem statement (Week 1); reviewed the annotated bibliography and conceptual framework for logical coherence and relevance (Week 2); approved the research design and methodology (Week 3); approved the validated instrument prior to deployment (Week 4); monitored data-collection integrity and dataset completeness (Weeks 5–6); verified the appropriateness of the analytical approach (Week 7); required and reviewed the interpretive depth of the Discussion section (Week 8); checked the coherence of the problem–method–results–conclusion chain across the full manuscript (Week 9); and confirmed the manuscript’s compliance readiness and the appropriateness of the submission venue (Weeks 10–11). Both authors provided final approval of the version to be published and agree to be accountable for all aspects of the work, including ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Disclosure statement

Funding

Institutional Review Board Statement

Data and Materials Availability

AI Declaration

The authors declare the use of Artificial Intelligence (AI) and digital research tools in the preparation of this manuscript. Specifically, ChatGPT (OpenAI) and Claude (Anthropic) were used to assist in refining academic writing, improving language clarity and coherence, organizing manuscript sections, strengthening methodological and statistical explanations, and enhancing responses to peer reviewers. QuillBot was used for language refinement, paraphrasing, grammar enhancement, and improving sentence readability. Google Forms was used for online questionnaire administration and data collection, Google Sheets for data organization, coding, and preliminary data preparation, Google Colab for statistical analyses and data visualization using Python-based libraries, and Mendeley Reference Manager for reference organization and citation management. All AI-generated outputs were carefully reviewed, verified, and edited by the authors. The authors assume full responsibility for the accuracy, originality, interpretation, integrity, and final content of the manuscript, and all scholarly judgments and conclusions remain entirely those of the authors.

Notes

Acknowledgement

Acknowledge all the people who have directly or indirectly contributed to the paper and mention their contribution. The authors acknowledge the Department of Computer Studies, College of Engineering, Architecture and Computing, Notre Dame of Marbel University, for institutional support during data collection; the ten expert validators who evaluated the KPI-Driven OJT Evaluation Instrument; and the BSCS and BSIT trainee-respondents of Cohorts I–III who participated in this study. 1. Doc. Hajah T. Sueno 2. Doc. Lumer Jude Doce 3. Ariel M. Mondejar 4. Susan M. Suarez 5. Jorge Nathaniel H. Amores 6. Aliah Chavy Sabado 7. Michael Paul Sebando 8. Abdulsalam O. Kusin 9. Romeo Rhyan G. Rapatan 10. Paolo Cajandig

References

Al Barwani, A. A., & Azam, S. M. F. (2023). Assessing the effect of OJT training quality factors on TVET students’ satisfaction in preparing for employment in Oman. European Journal of Education Studies, 10(8). https://doi.org/10.46827/ejes.v10i8.4910

Andrade, C. (2022). Research design: Cohort studies. Indian Journal of Psychological Medicine, 44(1), 93–97. https://doi.org/10.1177/02537176211073764

Badenes-Ribera, L., Silver, N. C., & Pedroli, E. (2020). Editorial: Scale development and score validation. Frontiers in Psychology, 11, Article 799. https://doi.org/10.3389/fpsyg.2020.00799

Bamber, C. (2023). Exploring enterprise-wide risk management system in higher education. Management Dynamics in the Knowledge Economy, 11(3), 267–285. https://doi.org/10.2478/mdke-2023-0017

Bamber, C., & Elezi, E. (2024). Enterprise-wide risk management in higher education: Beyond the paradigm of managing risk by categories. Business Process Management Journal. https://doi.org/10.1108/BPMJ-05-2024-0409

Barbera, J., Naibert, N., Komperda, R., & Pentecost, T. C. (2021). Clarity on Cronbach’s alpha use. Journal of Chemical Education, 98(2), 257–258. https://doi.org/10.1021/acs.jchemed.0c00183

Bartlett, M. S. (1954). A note on the multiplying factors for various chi square approximations. Journal of the Royal Statistical Society: Series B (Methodological), 16(2), 296–298. https://doi.org/10.1111/j.2517-6161.1954.tb00174.x

Bouwmans, M., Lub, X., Orlowski, M., & Nguyen, T.-V. (2024). Developing the digital transformation skills framework: A systematic literature review approach. PLoS ONE, 19(7), e0304127. https://doi.org/10.1371/journal.pone.0304127

Braun, V., & Clarke, V. (2022). Thematic analysis: A practical guide. SAGE Publications.

Bujang, M. A., Omar, E. D., & Baharum, N. A. (2018). A review on sample size determination for Cronbach’s alpha test: A simple guide for researchers. The Malaysian Journal of Medical Sciences, 25(6), 85–99. https://doi.org/10.21315/mjms2018.25.6.9

Calabit, M. C., & Paglinawan, J. L. (2024). Enhancing technical competency in higher education on-the-job training: Curriculum alignment, educational approach, and school governance execution. International Journal for Multidisciplinary Research (IJFMR), 6(3), Article IJFMR240321370. https://doi.org/10.36948/ijfmr.2024.v06i03.21370

Casugay, J. V. R., Cabauatan, W. F., Esperanza, J. A., Fontanilla, M. A. G., & Subang, A. H. (2024). Information technology students’ insights from their on-the-job training: Input to an internship manual. International Journal of Multidisciplinary Applied Business and Education Research, 5(2), 681–695. https://doi.org/10.11594/ijmaber.05.02.27

CC2020 Task Force. (2020). Computing curricula 2020: Paradigms for global computing education. Association for Computing Machinery & IEEE Computer Society. https://doi.org/10.1145/3467967

Chans, G. M., Alcalá, M. A., & Morales-Menendez, R. (2024). Future skills for Industry 4.0 integration and innovative learning for continuing engineering education. Frontiers in Education, 9, Article 1412018. https://doi.org/10.3389/feduc.2024.1412018

Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7(3), 309-319. https://doi.org/10.1037/1040-3590.7.3.309

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates. https://doi.org/10.4324/9780203771587

Commission on Higher Education. (2015). Revised policies, standards and guidelines for the Bachelor of Science in Information Technology (BSIT). CHED Memorandum Order No. 25, series of 2015. CHED.

Commission on Higher Education. (2017a). Policies, standards, and guidelines for the Bachelor of Science in Computer Science (BSCS) program. CHED Memorandum Order No. 25, series of 2017. CHED.

Commission on Higher Education. (2017b). Revised guidelines for student internship program in the Philippines. CHED Memorandum Order No. 104, series of 2017. CHED. https://ofa.upd.edu.ph/wp-content/uploads/2018/04/CMO-No.104-S.2017.pdf

Committee of Sponsoring Organizations of the Treadway Commission. (2017). Enterprise risk management — Integrating with strategy and performance. COSO. https://www.coso.org/SitePages/Guidance-on-Enterprise-Risk-Management.aspx

Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). SAGE.

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/BF02310555

Di Pietro, G. (2022). International internships and skill development: A systematic review. Review of Education, 10(1), e3363. https://doi.org/10.1002/rev3.3363

Edelsbrunner, P. A., Simonsmeier, B. A., & Schneider, M. (2025). The Cronbach’s alpha of domain-specific knowledge tests before and after learning: A meta-analysis of published studies. Educational Psychology Review, 37, Article 4. https://doi.org/10.1007/s10648-024-09982-y

Engku Abdullah, E., Yakob, R., & Bangaan Abdullah, M. (2025). A systematic review of enterprise risk management on higher education institutions’ performance. Open Education Studies, 7(1), Article 20250108. https://doi.org/10.1515/edu-2025-0108

Franco-Ángel, M., Carabali, J., & Velasco, M. I. (2023). The internship performance of undergraduate students: Are hard or soft skills more important? Industry and Higher Education, 37(3), 384–396. https://doi.org/10.1177/09504222221127213

Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2019). Multivariate data analysis (8th ed.). Cengage Learning.

Hair, J. F., Henseler, J., Dijkstra, T. K., & Sarstedt, M. (2022). A primer on partial least squares structural equation modeling (PLS-SEM) (3rd ed.). Sage. https://doi.org/10.1007/978-3-030-80519-7

Harman, H. H. (1976). Modern factor analysis (3rd ed.). University of Chicago Press.

Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., Kern, R., Picus, M., Hoyer, S., van Kerkwijk, M. H., Brett, M., Haldane, A., del Río, J. F., Wiebe, M., Peterson, P., … Oliphant, T. E. (2020). Array programming with NumPy. Nature, 585(7825), 357–362. https://doi.org/10.1038/s41586-020-2649-2

Hernández-Campos, M., Gonzalez-Torres, A., & García-Peñalvo, F. J. (2025). Learning outcomes evaluation through learning analytics systems in higher education: A systematic literature review. SAGE Open, 15(3). https://doi.org/10.1177/21582440251347374

Hernandez-de-Menendez, M., Escobar Díaz, C. A., & Morales-Menendez, R. (2020a). Engineering education for smart 4.0 technology: A review. International Journal on Interactive Design and Manufacturing (IJIDeM), 14(3), 789–803. https://doi.org/10.1007/s12008-020-00672-x

Hernandez-de-Menendez, M., Morales-Menendez, R., Escobar, C. A., & McGovern, M. (2020b). Competencies for Industry 4.0. International Journal on Interactive Design and Manufacturing (IJIDeM), 14(4), 1511-1524. https://doi.org/10.1007/s12008-020-00716-2

Holtom, B., Baruch, Y., Aguinis, H., & Ballinger, G. A. (2022). Survey response rates: Trends and a validity assessment framework. Human Relations, 75(8), 1560-1584. https://doi.org/10.1177/00187267211070769

IT and Business Process Association of the Philippines. (2022). Philippine IT-BPM industry roadmap 2028. IBPAP. https://admin.ibpap.org/storage/hub-resources/1fntWJGQKvZg4CfwP4yfXs0Jr1n85DVR1vYXJvSn.pdf

Kaiser, H. F. (1974). An index of factorial simplicity. Psychometrika, 39(1), 31–36. https://doi.org/10.1007/BF02291575

Kaltsonoudi, K., Tsigilis, N., & Karteroliotis, K. (2022). Common method bias in research designs using self-report tools: Literature overview and recommended remedies. Psychology: The Journal of the Hellenic Psychological Society, 27(1), 210–231. https://doi.org/10.12681/psyhps.30843

Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and development. Prentice-Hall.

Kolb, A. Y., & Kolb, D. A. (2022). Experiential Learning Theory as a guide for experiential educators in higher education. Experiential Learning and Teaching in Higher Education, 1(1), 38. https://doi.org/10.46787/elthe.v1i1.3362

Krejcie, R. V., & Morgan, D. W. (1970). Determining sample size for research activities. Educational and Psychological Measurement, 30(3), 607–610. https://doi.org/10.1177/001316447003000308

Li, L. (2022). Reskilling and upskilling the future-ready workforce for Industry 4.0 and beyond. Information Systems Frontiers, 26(5), 1697–1712. https://doi.org/10.1007/s10796-022-10308-y

Liang, J., Stephens, J. M., & Brown, G. T. L. (2025). A systematic review of the early impact of artificial intelligence on higher education curriculum, instruction, and assessment. Frontiers in Education, 10, Article 1522841. https://doi.org/10.3389/feduc.2025.1522841

Maheshwari, A. (2014). Data analytics made accessible. Amazon Digital Services.

Memon, M. A., Ting, H., Cheah, J.-H., Thurasamy, R., Chuah, F., & Cham, T. H. (2020). Sample size for survey research: Review and recommendations. Journal of Applied Structural Equation Modeling, 4(2), i–xx. https://doi.org/10.47263/JASEM.4(2)01

Miah, M. T., Erdei-Gally, S., Dancs, A., & Fekete-Farkas, M. (2024). A systematic review of Industry 4.0 technology on workforce employability and skills: Driving success factors and challenges in South Asia. Economies, 12(2), 31. https://doi.org/10.3390/economies12020031

Musa, S., Nurhayati, S., & Boriboon, G. (2025). The effect of internships on graduates’ employability, soft skills, and digital competence. Educational Process: International Journal (EDUPIJ). https://doi.org/10.22521/edupij.2025.17.306

Philippine Health Research Ethics Board. (2017). National ethical guidelines for health and health-related research (2017 ed.). Department of Science and Technology — Philippine Council for Health Research and Development. https://ethics.healthresearch.ph/

Podsakoff, P. M., Podsakoff, N. P., Williams, L. J., Huang, C., & Yang, J. (2024). Common method bias: It’s bad, it’s complex, it’s widespread, and it’s not easy to fix. Annual Review of Organizational Psychology and Organizational Behavior, 11, 17–61. https://doi.org/10.1146/annurev-orgpsych-110721-040030

Republic of the Philippines. (2012). Data Privacy Act of 2012 (Republic Act No. 10173). Official Gazette. https://www.officialgazette.gov.ph/2012/08/15/republic-act-no-10173/

Román-García, S., Almansa-Martínez, A., & Cruz-Díaz, M. R. (2024). Exploring digital competencies in higher education: Design and validation of instruments for the era of Industry 5.0. Frontiers in Education, 9, Article 1415800. https://doi.org/10.3389/feduc.2024.1415800

Santillana, P. M. P. (2026). Proposed quality internship framework for host training establishments (HTEs) (UP CIDS Discussion Paper Series 2026-04). University of the Philippines Center for Integrative and Development Studies. https://cids.up.edu.ph/wp-content/uploads/2026/03/Proposed-Quality-Internship-Framework-for-Host-Training-Establishments-HTEs.pdf

Schwab, K. (2017). The fourth industrial revolution. Crown Business.

Teshome, S. W. (2025). Key performance indicators in higher education: A systematic review of the conceptualization, dimensions, frameworks, and challenges of KPIs. International Journal of Education, Management, and Technology, 3(2), 611–623. https://doi.org/10.58578/ijemt.v3i2.6493

Ülker, N., & Otrar, M. (2025). University 4.0 essentials: Influence of Industry 4.0 on higher education. Industry and Higher Education, 39(5), 528–538. https://doi.org/10.1177/09504222241295625

Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S. J., Brett, M., Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E., … SciPy 1.0 Contributors. (2020). SciPy 1.0: Fundamental algorithms for scientific computing in Python. Nature Methods, 17(3), 261–272. https://doi.org/10.1038/s41592-019-0686-2

Vuorikari, R., Pokropek, A., & Castaño Muñoz, J. (2026). Enhancing digital skills assessment: Introducing compact tools for measuring digital competence. Technology, Knowledge and Learning, 31, 519–546. https://doi.org/10.1007/s10758-025-09825-x

Wu, M.-J., Zhao, K., & Fils-Aime, F. (2022). Response rates of online surveys in published research: A meta-analysis. Computers in Human Behavior Reports, 7, Article 100206. https://doi.org/10.1016/j.chbr.2022.100206

Yan, J., Tian, H., Sun, X., & Song, L. (2025). Role of artificial intelligence in enhancing competency assessment and transforming curriculum in higher vocational education. Frontiers in Education, 10, Article 1551596. https://doi.org/10.3389/feduc.2025.1551596

Zakariya, Y. F. (2022). Cronbach’s alpha in mathematics education research: Its appropriateness, overuse, and alternatives in estimating scale reliability. Frontiers in Psychology, 13, Article 1074430. https://doi.org/10.3389/fpsyg.2022.1074430

Zamora Iribarren, M., Garay-Rondero, C. L., Lemus-Aguilar, I., & Peimbert-García, R. E. (2024). A review of Industry 4.0 assessment instruments for digital transformation. Applied Sciences, 14(5), 1693. https://doi.org/10.3390/app14051693

Cite this article:

Balala, B.M. & Yu, E.R. (2026). A risk-based audit framework for performance-driven on-the-job training quality in computing education: Evidence from a three-year sequential cohort study. International Journal of Educational Management and Development Studies, 7(3), 27-25. https://doi.org/10.53378/ijemds.353373