EVALUATING LLM-BASED AI SYSTEMS ON THE POLISH MATHEMATICS MATURA EXAMINATION: IMPLICATIONS FOR AI GOVERNANCE AND QUALITY ASSURANCE IN EDUCATIONAL ORGANIZATIONS
DOI:
https://doi.org/10.17512/znpcz.2026.3.10Keywords:
AI governance, error analysis, LLM-based AI systems, mathematical reasoning, mathematics education, quality assuranceAbstract
LLM-based AI systems are increasingly used in education, including as tools for solv-ing, explaining, and checking mathematical tasks. This exploratory study evaluates the performance of four widely used user-facing AI systems – ChatGPT, Claude, Gemini, and Perplexity – on the Polish mathematics Matura examination at the basic and extended levels. The systems were tested through their chat interfaces using official CKE examination papers, submitted as complete attachments, and a single fixed prompt for each level. The responses were assessed according to the official CKE marking schemes, with points awarded for the final responses and for the correctly completed stages of open-ended tasks. All systems achieved high scores at the basic level (86-96%), whereas performance at the extended level varied considerably (38-100%). At the basic level, most lost points resulted from errors in interpreting charts, figures or other visual information. At the extended level, errors were more often associated with mathematical reasoning or methods. The findings demonstrate the value of supplementing aggregate examination scores with diagnostic error profiles that distinguish visual-data processing from mathematical reasoning. For educational organizations, such benchmarks may provide one source of evidence for human oversight, risk identification and periodic quality assessment. However, the results reflect the performance of complete AI systems under specific interface conditions and should not be interpreted as direct evidence of their general educational effectiveness. Further studies involving multiple examination papers, repeated trials and educational users are needed.
References
Ahn, J., Verma, R., Lou, R., Liu, D., Zhang, R., & Yin, W. (2024). Large language models for mathematical reasoning: Progresses and challenges. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, 225-237. https://doi.org/10.18653/v1/2024.eacl-srw.17
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., Arx, S. V., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N. S., Chen, A. S., Creel, K. A., Davis, J., Demszky, D., … Liang, P. (2021). On the opportunities and risks of foundation models. arXiv. https://arxiv.org/abs/2108.07258
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., … Amodei, D. (2020). Language models are few-shot learners. arXiv. https://doi.org/10.48550/arXiv.2005.14165
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T.,
Li, Y., Lundberg, S., Nori, H., Palangi, H., Ribeiro, M., & Zhang, Y. (2023). Sparks of artifi-cial general intelligence: Early experiments with GPT-4. arXiv. https://doi.org/10.48550/arXiv.2303.12712
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J.,
Hilton, J., Nakano, R., Hesse, C., & Schulman, J. (2021). Training verifiers to solve math word problems. arXiv. https://arxiv.org/abs/2110.14168
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of infor-mation technology. MIS Quarterly, 13(3), 319-340. https://doi.org/10.2307/249008
Dong, Y., Jiang, X., Liu, H., Jin, Z., & Li, G. (2024). Generalization or memorization: Data con-tamination and trustworthy evaluation for large language models. arXiv. https://arxiv.org/abs/2402.15938
Dwivedi, Y. K., Hughes, L., Ismagilova, E., Aarts, G., Coombs, C., Crick, T., Duan, Y., Dwivedi, R., Edwards, J., Eirug, A., Galanos, V., Ilavarasan, P. V., Janssen, M., Jones, P., Kar, A. K.,
Kizgin, H., Kronemann, B., Lal, B., Lucini, B., … Williams, M. D. (2021). Artificial intelli-gence (AI): Multidisciplinary perspectives on emerging challenges, opportunities, and agen-da for research, practice and policy. International Journal of Information Management, 57, Article 101994. https://doi.org/10.1016/j.ijinfomgt.2019.08.002
Floridi, L., Cowls, J., Beltrametti, M., Chatila, R., Chazerand, P., Dignum, V., Luetge, C., Madelin, R., Pagallo, U., Rossi, F., Schafer, B., Valcke, P., & Vayena, E. (2018). AI4People—An eth-ical framework for a good AI society: Opportunities, risks, principles, and recommenda-tions. Minds and Machines, 28, 689-707. https://doi.org/10.1007/s11023-018-9482-5
Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., … Zhang. Z. (2025). DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645, 633-638. https://doi.org/10.1038/s41586-025-09422-z
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D. X., & Steinhardt, J. (2020). Measuring massive multitask language understanding. arXiv. https://arxiv.org/abs/2009.03300
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D. X., & Steinhardt, J. (2021). Measuring mathematical problem solving with the MATH dataset. arXiv. https://arxiv.org/abs/2103.03874
Heung, Y., & Chiu, T. K. F. (2025). How ChatGPT impacts student engagement from a systematic review and meta-analysis study. Computers and Education: Artificial Intelligence. https://doi.org/10.1016/j.caeai.2025.100361
Jassem, K., Ciesiólka, M., Graliński, F., Jabłoński, P., Pokrywka, J., Kubis, M., Jabłońska, M.,
& Staruch, R. (2025). LLMzSzŁ: A comprehensive LLM benchmark for Polish. arXiv. https://arxiv.org/abs/2501.02266
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020). Scaling laws for neural language models. arXiv. https://arxiv.org/abs/2001.08361
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G. L., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., … Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for educa-tion. Learning and Individual Differences, 103, Article 102274. https://doi.org/10.1016/j.lindif.2023.102274
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022). Large language models are zero-shot reasoners. arXiv. https://arxiv.org/abs/2205.11916
Kotter, J. P. (1996). Leading change. Harvard Business School Press.
Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., Wu, Y., Neyshabur, B., Gur-Ari, G., & Misra, V. (2022). Solving quantitative reasoning problems with language models. arXiv. https://doi.org/10.48550/arXiv.2206.14858
Mirzadeh, I., Alizadeh-Vahid, K., Shahrokhi, H., Tuzel, O., Bengio, S., & Farajtabar, M. (2024). GSM-Symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv. https://arxiv.org/abs/2410.05229
Nonaka, I., & Takeuchi, H. (1995). The knowledge-creating company: How Japanese companies
create the dynamics of innovation. Oxford University Press. https://doi.org/10.1093/oso/9780195092691.001.0001
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D.,
Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V.,
Baltescu, P., Bao, H., Bavarian, M., Belgum, J., … Zoph, B. (2023). GPT-4 technical report. arXiv. https://arxiv.org/abs/2303.08774
OpenAI, Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., Iftimie, A., Karpenko, A., Passos, A. T., Neitz, A., Pro-kofiev, A., Wei, A., Allison Tam, Bennett, A., … Li, Z. (2024). OpenAI o1 system card. arXiv. https://arxiv.org/abs/2412.16720
Parasuraman, A., Zeithaml, V., & Berry, L. L. (1988). SERVQUAL: A multiple-item scale for measuring consumer perceptions of service quality. Journal of Retailing, 64(1), 12-40.
Pardos, Z. A., & Bhandari, S. (2024). ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills. PLOS One, 19, Article e0304013. https://doi.org/10.1371/journal.pone.0304013
Pokrywka, J., Kaczmarek, J., & Gorzelańczyk, E. (2024). GPT-4 passes most of the 297 written Polish board certification examinations. SSRN. https://doi.org/10.2139/ssrn.4861528
Pyeon, G., Heo, I., Jung, J., Hwang, T., Namgoong, H., Seo, H., Han, Y., Kim, E., Kang, H., & Jung, S. (2025). Evaluating large language models on the 2026 Korean CSAT mathematics exam: Measuring mathematical ability in a zero-data-leakage setting. arXiv. https://arxiv.org/abs/2511.18649
Rogers, E. M. (2003). Diffusion of innovations (5th ed.). Free Press.
Rosoł, M., Gąsior, J. S., Łaba, J., Korzeniewski, K., & Młyńczak, M. (2023). Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish Medical Final Examination. Scientific Re-ports, 13, Article 20512. https://doi.org/10.1038/s41598-023-46995-z
Setälä, M., Sikström, P., Heilala, V., & Kärkkäinen, T. (2025). Assessment of evolving large lan-guage models in upper secondary mathematics. Proceedings of ICETC 2025, 210-214. https://doi.org/10.1109/ICETC66579.2025.11387533
Turmuzi, M., Azmi, S., & Kertiyani, N. M. (2026). ChatGPT in school mathematics education: A systematic review of opportunities, challenges, and pedagogical implications. Teaching and Teacher Education, 170, Article 105286.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Po-losukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems. https://arxiv.org/abs/1706.03762
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., & Fedus, W. (2022b). Emergent abilities of large language models. arXiv. https://arxiv.org/abs/2206.07682
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E. H., Xia, F., Le, Q., & Zhou, D. (2022a). Chain-of-thought prompting elicits reasoning in large language models. arXiv. https://arxiv.org/abs/2201.11903
Zhao, W., Zhou, K., Junyi, L., Tianyi, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., Liu, Z., … Wen, J. R. (2026). A survey of large language models. Frontiers of Computer Science, 20, Article 2012627. https://doi.org/10.1007/s11704-026-60308-3
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Maciej Renz, Grzegorz Borkowski, Ewelina Ząb (Autor)
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors retain copyright and grant the journal the right of first publication, with the work simultaneously licensed under a Creative Commons Attribution 4.0 International License, which permits others to share the work with appropriate acknowledgment of the authorship and initial publication in this journal.
