Cohen, A. Designing and evaluating standard-setting procedures for licensure and certification tests. Advances in Health Sciences Education, 4, — Cronbach, L. Construct validity in psychological tests. Psychological Bulletin, 52, — Ebel, R. Essentials of educational measurement. Engelwood Cliffs: Prentice Hall. Embretson, S. Item response theory for psychologists. Mahwah: Lawrence Erlbaum Associates.
Freunberger, R. Standard-Setting Mathematik 8. Technischer Bericht [Standard setting in mathematics Grade 8. Technical Report]. Haertel, E. Standard setting as a participatory process: Implications for validation of standards-based accountability programs. Educational measurement: Issues and practice, 21 1 , 16— Hahn, I. Assessing scientific literacy over the lifespan: A description of the NEPS science framework and the test development.
Journal of Educational Research Online, 5 2 , — Hambleton, R. Newspaper coverage of NAEP results to Byrd Eds. Hsieh, Mingchuan. Language Assessment Quarterly, 10, — Hurtz, G. A meta-analysis of the effects of modifications to the Angoff method on cutoff scores and judgement consensus.
Educational and Psychological Measurement, 63, — Impara, J. Standard setting: An alternative approach. Journal of Educational Measurement, 34, — Kane, M.
Validating the performance standards associated with passing scores. Review of Educational Research, 64, — Terminology, emphasis, and utility in validation. Educational Researcher, 37, 76— Validating the interpretations and uses of test scores.
Journal of Educational Measurement, 50, 1— Landis, J. The measurement of observer agreement for categorical data. Biometrics, 33, — Lane, S. Strategies for examining the consequences of assessment and accountability programs.
Educational measurement: Issues and practice, 21 1 , 23— Leucht, M. Leucht, N. Fachleistungen beim Abitur: Vergleich allgemeinbildender und beruflicher Gymnasien in Schleswig-Holstein [Abilities at the end of upper secondary education. A comparison between academic and vocational upper secondary schools in Schleswig-Holstein]. Lissitz, R. A suggested change in terminology and emphasis regarding validity and education. Educational Researcher, 36, — Massey, A.
Educational and Psychological Measurement, 57, — McGinty, D. Applied Measurement in Education, 18, — Messick, S. The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23 2 , 13— Validity of psychological assessment.
American Psychologist, 50, — Organisation for Economic Co-operation and Development. PISA Technical report. Pant, H. Validity issues in standard-setting studies. Studies in Educational Evaluation, 35, 95— Competencies in mathematics and science at the end of secondary level I]. Parker, P. Plake, B. Cizek Ed. Foundations, methods, and innovations pp. New York: Routledge. Shephard, L. Setting performance standards for student achievement. Stanford: National Academy of Education. Sireci, S. Evaluation of the standard setting on the grade 12 national assessment of educational progress mathematics test.
Applied Measurement in Education, 22, — Skaggs, G. Educational and Psychological Measurement, 71, — Stanat, P. IQB trends in student achievement The second national assessment of language proficiency at the end of the ninth grade.
Tiffin-Richards, S. Setting standards for English foreign language assessment: Methodology, validation, and a degree of arbitrariness. Educational Measurement: Issues and Practice. Wang, M.
Who chooses STEM careers? Using a relative cognitive strength and interest model to predict careers in science, technology, engineering, and mathematics. Journal of Youth and Adolescence, 46, — Wu, Y. A multivariate generalizability theory approach to standard setting. Applied Psychological Measurement, 39, — Yousuf, N.
Teaching and Learning in Medicine, 27, — Yudkowski, R. Teaching and Learning in Medicine, 20, — Download references. NK carried out the standard setting procedure as well a the validation study. She wrote the different drafts including the final one. HW performed the G studies and provided feedback on the different drafts of the manuscript. OK provided feedback on the different drafts of the manuscript. All authors read and approved the final manuscript. The data of the student LSA has not yet been made available to the public.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. You can also search for this author in PubMed Google Scholar. Correspondence to Nele Kampa. See Table 7. See Figs. Ratings of the experts for each cut score by Angoff method, round 1. Values of the same item format share the same color, values of the same expert share the same marker.
MC multiple choice items, CMC complex multiple choice items. Ratings of the experts for each cut score by Angoff method, round 2.
MC multiple choice items, CMC scomplex multiple choice items. Reprints and Permissions. Kampa, N. The standard setting process: validating interpretations of stakeholders. Large-scale Assess Educ 7, 3 Download citation. Received : 08 June Accepted : 12 February Published : 18 February Anyone you share the following link with will be able to read this content:.
Sorry, a shareable link is not currently available for this article. Provided by the Springer Nature SharedIt content-sharing initiative. Skip to main content. Search all SpringerOpen articles Search. Download PDF. Results Our t-tests and G-studies showed that the intended use of the cut scores was valid regarding procedural and internal aspects of validity. Introduction Most large-scale assessments LSA aim to provide findings on the effectiveness of a school system to different stakeholders e.
The concept of validity The concept of validity underwent major changes. Table 1 Evaluation elements and aspects of standard-setting procedures. Full size table. Method Study design and participants The LISA 6-Study Educatioal Outcomes of Students from Vocational and Academic Upper Secondary Schools-Study was carried out in one federal state of Germany in order to provide information to stakeholders about the abilities of students in upper secondary level education in mathematics, science and English first foreign language.
Statistical analyses To support the procedural validity , we report the means of answers on the questionnaires regarding clarity and explicitness of the process P 2 , familiarity with the process P 3 , and confidence in the resulting cut scores P 4.
Full size image. Table 2 Cut scores of standard-setting procedure average number of items solved at border of a cut score Full size table. Table 4 Science ability cut score VCs and G coefficients of the reliability analyses Full size table. Table 5 Multinomial regression of number of lessons per week on PL Full size table.
Table 6 Effects of PL placement and ability in science and mathematics on studies in science Full size table. Discussion In our study, we investigated validation arguments that derived from a standard-setting procedure based on two variations of the Angoff method Angoff Procedural validation The description of the expert panel for the standard-setting showed that the experienced experts represent the diversity of the field P 1.
Internal validation Our G studies showed that the experts made cut score recommendations across the two Angoff variations, of which the reliability was between 0. External validation To gain insights into the interpretation of the science PL, we calculated four consecutive multinomial logistic regressions that included numbers of lessons per week in science, mathematics, and German, as well as school type.
Consequential validation Evidence of the consequential validation provided valuable indications for how to interpret the PL and, maybe more importantly, for how not to interpret them. Overall evaluation of the validity Taking a look at the comprehensive body of validity evidence for the interpretation of the cut scores that derived from our standard-setting, we conclude that our validity study was fruitful in several respects. Notes 1. Google Scholar Angoff, W. Google Scholar Arrasmith, D. Google Scholar Bloch, R.
Article Google Scholar Brennan, R. Book Google Scholar Bullock, C. Article Google Scholar Cizek, G. Google Scholar Cohen, A. Article Google Scholar Cronbach, L. Article Google Scholar Ebel, R. Google Scholar Embretson, S. Google Scholar Freunberger, R. Article Google Scholar Hahn, I. Google Scholar Hambleton, R. Google Scholar Hsieh, Mingchuan. Article Google Scholar Hurtz, G. Article Google Scholar Impara, J.
Article Google Scholar Kane, M. Article Google Scholar Landis, J. Article Google Scholar Lane, S. Article Google Scholar Leucht, M. Google Scholar Leucht, M. Google Scholar Lissitz, R. Article Google Scholar Massey, A. Article Google Scholar Messick, S. Google Scholar Pant, H. Article Google Scholar Pant, H. Google Scholar Parker, P. Article Google Scholar Plake, B. Google Scholar Shephard, L.
Google Scholar Sireci, S. Article Google Scholar Skaggs, G. Article Google Scholar Stanat, P. Article Google Scholar Wang, M. Article Google Scholar Wu, Y. Article Google Scholar Yousuf, N. Article Google Scholar Yudkowski, R. Article Google Scholar Download references. Competing interests The authors declare that they have no competing interests.
Adoption and copyright. News and resources. IFRS Translations. Editorial corrections. IFRS Taxonomy. Supporting consistent application. Work plan. Post-implementation Reviews. Pipeline projects. Open for comment. Better Communication in Financial Reporting.
We support the development, adoption, and implementation of high-quality international standards. We work to prepare a future-ready accounting profession. We speak out as the voice of the global accounting profession. Thank you for your interest in our publications. These valuable works are the product of substantial time, effort and resources, which you acknowledge by accepting the following terms of use.
You may not reproduce, store, transmit in any form or by any means, with the exception of non-commercial use e. Our reproduction and translation policies, as well as our online permission request and inquiry system, are accessible on the Permissions Information web page. For additional information, please read our website Terms of Use.
0コメント