Welcome to the new AACRAO website. While the site is live, development is ongoing, visitors can expect updates and new features in the weeks ahead.

Mobile Nav
Log in
Join
Get Involved

You must be logged in to view this content. Please login or learn more about joining AACRAO. If you have any questions or experience any issues logging in, please reach out to us atmembership@aacrao.org.

Log In Join

Topics

,

Article Type

Professional Proficiency

,

Article Abstract

Related Issue:

While college readiness continues to dominate the educational landscape in the United States, students still leave high school not ready for college. As a consequence, admissions officers need to consider all available indicators (e.g., high school performance, admissions test scores, college preparatory courses) in order to effectively evaluate prospective student applications. Predictive validity studies are one of the inherent mechanisms in the admission process and can be used to predict college performance and success. These studies ultimately are guided by the availability of data, which can be challenging to identify and procure. Although predictive validity research has played an important role in informing admissions policy in recent years, few published works appear to include a full, informative account of its role in helping to establish college readiness and the importance of identifying different sources of data for doing so. This paper aims to fill that gap, bringing together a detailed explanation of the role of predictive validity studies to investigate college readiness and success and the identification of “fit-for-purpose” data to carry out such studies and inform postsecondary admissions policy and decisions.

Author(s)

shaw@2xStuart Shaw is the Principal Research Officer for Cambridge Assessment International Education, Cambridge Assessment. Before leading a research team in the area of mainstream international examinations, Shaw worked on a range of Cambridge English products with specific skill responsibilities for writing. He is particularly interested in demonstrating how Cambridge Assessment seeks to meet the demands of validity in its tests.

rodeiro@2xCarmen Vidal Rodeiro is a Senior Research Officer at Cambridge Assessment. Her main areas of interests include subject provision and uptake in schools and colleges, comparability of standards in high-stakes examinations, progression routes and predictors of university participation, and success both in the UK and internationally. She has a Ph.D. in statistics from the University of Aberdeen.

References

ACT. 2016. The Condition of College and Career Readiness 2016. Iowa City, IA: ACT, Inc.

ACT. 2017. National Collegiate Retention and Persistence-to-Degree Rates. Iowa City, IA: ACT, Inc.

AERA. See American Educational Research Association, American Psychological Association, and National Council on Measurement in Education.

Alderson, J. C., C. Clapham, and D. Wall. 1995. Language Test Construction and Evaluation. Cambridge: Cambridge University Press.

American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. 2014. Standards for Educational and Psychological Testing. Washington, D.C.: American Educational Research Association.

Anastasi, A. 1988. Psychological Testing (6th edition). New York: Macmillan.

Attewell, P., D. Lavin, T. Domina, and T. Levey. 2006. New evidence on college remediation. Journal of Higher Education. 77(5): 886–924.

Banerjee, J. V. 2003. Interpreting and using proficiency test scores. Unpublished Ph.D. thesis. Lancaster: University of Lancaster.

Belfield, C. R., and P. M. Crosta. 2012. Predicting Success in College: The Importance of Placement Tests and High School Transcripts. New York: Community College Research Center, Columbia University.

Bell, J. F., and T. Dexter. 2000. Using multilevel models to assess the comparability of examinations. Paper presented at the 5th International Conference on Social Science Methodology, October 2000.

Bergeron, L. 2015. Diploma Programme Students’ Enrollment and Outcomes at U.S. Postsecondary Institutions 2008–2014. A research report prepared for the International Baccalaureate. LaCrosse, WI: University of Wisconsin La Crosse.

Bettinger, E., and R. Baker. 2011. The Effects of Student Coaching in College: An Evaluation of a Randomized Experiment in Student Mentoring. Cambridge, MA: National Bureau of Economic Research.

Bettinger, E., and B. T. Long. 2004. Shape Up or Ship Out: The Effects of Remediation on Students at Four-Year Colleges. Cambridge, MA: National Bureau of Economic Research.

Burton, N.W., and L. Ramist. 2001. Predicting Success in College: SAT Studies of Classes Graduating Since 1980. New York: The College Board.

Camara, W. 2013. Defining and measuring college and career readiness: a validation framework. Educational Measurement: Issues and Practice. 32(4): 16–27.

Chen, X., and S. Simone. 2016. Remedial Coursetaking at U.S. Public 2- and 4-Year Institutions: Scope, Experiences and Outcomes. Washington, D.C.: National Center for Education Statistics, U.S. Department of Education.

Clinedinst, M. E., S. F. Hurley, and D. A. Hawkins. 2013. State of College Admission 2013. Washington, D.C.: National Association for College Admission Counseling.

Clinedinst, M.E., and A. M. Koranteng. 2017. State of College Admission 2017. Washington, D.C.: National Association for College Admission Counseling.

Clinedinst, M.E., A. M. Koranteng, and T. Nicola. 2015. State of College Admission 2015. Washington, D.C.: National Association for College Admission Counseling.

Coca, V., T. Johnson, T. Kelley-Temple, M. Roderick, E. Moeller, N. Williams, and K. Moragne. 2012. Working to My Potential: The Postsecondary Experiences of CPS Students in the International Baccalaureate Diploma Programme. Chicago: The University of Chicago.

Conley, D. T. 2012. College and career readiness: Same or different? Educational Leadership. 69(7): 29–34.

Conley, D. T., K. V. Drummond, A. de Gonzalez, J. Rooseboom, and O. Stout. 2011. Reaching the Goal: The Applicability and Importance of the Common Core State Standards to College and Career Readiness. Portland, OR: Educational Policy Improvement Center.

Conley, D., C. McGaughy, W. Davis-Molin, R. Farkas, and E. Fukuda. 2014. International Baccalaureate Diploma Programme: Examining College Readiness. Bethesda, MD: International Baccalaureate Organization.

Cresswell, J. W., and V. L. Plano Clark. 2011. Designing and Conducting Mixed Methods Research. Thousand Oaks, CA: Sage Publications.

Data Quality Campaign. 2005. Creating a Longitudinal Data System: Using Data to Improve Student Achievement. Austin, TX: Data Quality Campaign.

Deming, D. J., J. S. Hastings, T. J. Kane, and D. O. Staiger. 2014. School choice, school quality and postsecondary attainment. American Economic Review. 104(3): 991–1013.

Denzin, N. 1978. The Research Act: A Theoretical Introduction to Sociological Methods. New York: McGraw-Hill.

Dunbar, D. 2007. What You Don’t Know Can Keep You Out of College. New York: Gotham Books.

Dynarski, S., J. Hyman, and D. W. Schanzanbach. 2013. Experimental evidence on the effect of childhood investments on postsecondary attainment and degree completion. Journal of Policy Analysis and Management, 32(4): 692–717.

Dynarski, S., S. W. Hemelt, and J. Hyman. 2013. The Missing Manual: Using National Student Clearinghouse Data to Track Postsecondary Outcomes. Washington, D.C.: Institute of Education Sciences, U.S. Department of Education.

FairTest. 2017. FairTest Reaction to Class of 2017 SAT Results. Jamaica Plain, MA: The National Center for Fair and Open Testing.

Farrington, C. A., M. Roderick, E. Allensworth, J. Nagaoka, T. S. Keyes, D. W. Johnson, and N. O. Beechum. 2012. Teaching Adolescents to Become Learners: The Role of Noncognitive Factors in Shaping School Performance—A Critical Literature Review. Chicago: Consortium on Chicago School Research.

Farruggia, S. P., C. Han, L. Watson, T. P. Moss, and B. L. Bottoms. 2016. Noncognitive factors and college student success. Journal of College Student Retention: Research, Theory & Practice. 20(3): 308–327.

Goldring, E., and M. Berends. 2009. Leading with Data: Pathways to Improve Your School. Thousand Oaks, CA: Sage Publishing.

Greene, J. C., V. L. Caracelli, and W. F. Graham. 1989. Toward a conceptual framework for mixed-method evaluation design. Educational Evaluation and Policy Analysis. 11(3): 255–274.

Hemelt, S. W., K. B. Roth, and W. W. Eaton. 2013. Elementary school interventions: experimental evidence in postsecondary outcomes. Educational Evaluation and Policy Analysis. 35(4): 413–436.

Hernandez, M. 2009. A Is for Admission: The Insider’s Guide to Getting into the Ivy League and Other Top Colleges. New York: Grand Central Publishing.

Hill, J., N. Smith, D. Wilson, J. Wine, and D. Richards. 2016. 2012/14 Beginning Postsecondary Students Longitudinal Study. Data File Documentation. Washington, D.C.: National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education.

Hyman, J. 2013. ACT for All: The Effect of Mandatory College Entrance Exams on Postsecondary Attainment and Choice. Ann Arbor, MI: University of Michigan.

Iatarola, P., D. Conger, and M. C. Long. 2011. Determinants of high schools’ advanced course offerings. Educational Evaluation and Policy Analysis. 33(3): 340–359.

IBO. See International Baccalaureate Organization.

Ingels, S. J., D. J. Pratt, C. P. Alexander, D. M. Jewell, E. Lauff, T. L. Mattox, and D. Wilson. 2014. Education Longitudinal Study of 2002: Third Follow-Up Data File Documentation. Washington, D.C.: National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education.

Ingels, S. J., D. J. Pratt, D. Herget, M. Bryan, L. B. Fritch, R. Ottem, J. E. Rogers, and D. Wilson. 2015. High School Longitudinal Study of 2009: 2013 Update and High School Transcript Data File Documentation. Washington, D.C.: National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education.

International Baccalaureate Organization. 2016. The IB Diploma Programme Statistical Bulletin, May 2016 Examination Session. Den Haag, Netherlands.

Kalsbeek, D., M. Sandlin, and W. Sedlacek. 2013. Employing Noncognitive Variables to Improve Admissions and Increase Student Diversity and Retention. Strategic Enrollment Management Quarterly. 1: 132–150.

Kena, G., L. Musu-Gillette, J. Robinson, X. Wang, A. Rathbun, J. Zhang, S. Wilkinson-Flicker, A. Barmer, and E. Dunlop Velez. 2015. The Condition of Education 2015 (NCES 2015-144). Washington, D.C.: U.S. Department of Education, National Center for Education Statistics.

Killgore, L. 2009. Merit and competition in selective college admissions. The Review of Higher Education, 32(4): 469–488.

Klopfenstein, K., and M. K. Thomas. 2009. The link between Advanced Placement experience and early college success. Southern Economic Journal. 75(3): 873–891.

Kobrin, J. L., B. F. Patterson, E. J. Shaw, K. D. Mattern, and S. M. Barbuti. 2008. Validity of the SAT® for Predicting First-Year College Grade Point Average. New York: The College Board.

Liu, L. O., D. M. Klieger, J. L. Bochenek, S. L. Holtzman, and J. Xu. 2016. An Investigation of the Use and Predictive Validity of Scores from the GRE® Revised General Test in a Singaporean University. Princeton, NJ: Educational Testing Service.

Makransky, G., P. Havmose, M. L. Vang, T. E. Andersen, and T. Nielsen. 2017. The predictive validity of using admissions testing and multiple mini-interviews in undergraduate university admissions. Higher Education Research & Development. 36(5).

Mattern K. D., J. P. Marini, and E. J. Shaw. 2013. Are AP Students More Likely to Graduate from College on Time? New York: The College Board.

Mattern, K. D., X. Xiong, and E. J. Shaw. 2009. The Relationship Between AP Exam Performance and College Outcomes. New York: The College Board.

Maruyama, G. 2012. Assessing college readiness: should we be satisfied with ACT or other threshold scores? Educational Researcher. 41: 252–261.

Messick, S. 1988. The once and future issues of validity: Assessing the meaning and consequences of measurement. In Test Validity, edited by H. Wainer and I. H. Braun. Hillsdale, NJ: Lawrence Erlbaum Associates.

Morgan, R., and R. Maneckshana. 2000. AP Students in College: An Investigation of Their Course-Taking Patterns and College Majors. Princeton, NJ: Educational Testing Service.

Morgan, R., and L. Ramist. 1998. Advanced Placement Students in College: An Investigation of Course Grades at 21 Colleges. Princeton, NJ: Educational Testing Service.

NACAC. See National Association of College Admissions Counseling.

National Association of College Admissions Counseling. 2007. NACAC Admission Trends Survey. Arlington, VA: NACAC.

National Association of College Admissions Counseling. 2016. NACAC Admission Trends Survey. Arlington, VA: NACAC.

National Student Clearinghouse. 2018. About the Clearinghouse. Herndon, VA: NSC.

NSC. See National Student Clearinghouse.

Noble, J. P., and R. Sawyer. 2002. Predicting Different Levels of Academic Success in College Using High School GPA and ACT Composite Score. Iowa City, IA: American College Testing.

Panich, C. 2001. A Study of the University Performance of Students With International Baccalaureate High School Experience. Pittsburgh, PA: Duquesne University.

Radford, A. W., and L. Horn. 2012. Web Tables: An Overview of Classes Taken and Credits Earned by Beginning Postsecondary Students. Washington, D.C.: National Center for Education Statistics, U.S. Department of Education.

Rigol, G. W. 2003. Admissions Decision-Making Models: How U.S. Institutions of Higher Education Select Undergraduate Students. New York: The College Board.

Roderick, M., J. Nagaoka, and E. Allensworth. 2006. From High School to the Future: A First Look at Chicago Public School Graduates’ College Enrollment, College Preparation and Graduation from Four-Year Colleges. Chicago: Consortium on Chicago School Research at the University of Chicago.

Rothstein, J. M. 2004. College performance prediction and the SAT. Journal of Econometrics. 121: 297–317.

Rubin, R. 2014. Who gets in and why? An examination of admissions to America’s most selective colleges and universities. International Education Research. 2(2): 1–18.

Sackett, P. R., and F. Lievens. 2008. Personnel selection. Annual Review of Psychology. 59: 419–450.

Schmitt, N., J. Keeney, F. L. Oswald, T. J. Pleskac, A. Q. Billington, R. Sinha, and M. Zorzie. 2009. Prediction of four-year college student performance using cognitive and noncognitive predictors and the impact on demographic status of admitted students. Journal of Applied Psychology. 94(6): 1470–1497.

Sedlacek, W. 2017. Measuring Noncognitive Variables: Improving Admissions, Success and Retention for Underrepresented Students. Sterling, VA: Stylus Publishing, LLC.

Shaw, S. D. 2001. Investigating the impact of Cambridge international assessments on U.S. stakeholders: student and teacher perceptions. College and University. 87(2): 12–23.

Shaw, S. D., and C. Bailey. 2011. Success in the US: Are Cambridge International assessments good preparation for university study? Journal of College Admission. 213: 6–16.

Shaw, S. D, J. Warren, and T. Gill. 2014. Assessing the Impact of the Cambridge International Acceleration Program on US University Determinants of Success: a Multilevel Modelling Approach. College and University. 89(4): 38-54.

Skomsvold, P. 2014. Profile of Undergraduate Students: 2011–12. Washington, D.C.: National Center for Education Statistics, U.S. Department of Education.

Symonds, W. C., R. B. Schwartz, and R. Ferguson. 2011. Pathways to Prosperity: Meeting the Challenge of Preparing Young Americans for the 21st Century. Boston: Pathways to Prosperity Project and Harvard Graduate School of Education.

The College Board. 2017. AP Program Size and Increments 1998–2017. New York: The College Board.

U.S. Department of Education. 2009. Implementing Data-Informed Decision Making in Schools: Teacher Access, Supports and Use. Washington, D.C.: U.S. Department of Education.

U.S. Department of Education. 2014. College and Career Readiness Issue Brief No. 3. Washington, D.C.: U.S. Department of Education.

U.S. Department of Education. 2015. Every Student Succeeds Act. Washington, D.C.: U.S. Department of Education.

Vidal Rodeiro, C., C. Crawford, and S. D. Shaw. 2017. From “AICE”-ing the test to earning the degree: enrollment and graduation patterns among students with the Cambridge Advanced International Certificate of Education (AICE) Diploma. College and University. 92(4): 12–23.

Weir, C. J. 2005. Language Testing and Validation: An Evidence-Based Approach. Basingstoke: Palgrave Macmillan.

Wiley, A., J. N. Wyatt, and W. J. Camara. 2010. The Development of an Index of College Readiness. New York: The College Board.

Williams, N. J., L. Keng, and K. O. O’Malley. 2012. Maximizing panel input: Incorporating empirical evidence in a way the standard setting panel will understand. Paper presented at the meeting of the National Council on Measurement in Education, April 14–16, 2012. Vancouver, Canada.

Yin, R. K. 2006. Case study methods. In Handbook of Complementary Methods in Education Research, edited by J. L. Green, G. Camilli, and P.B. Emore. Washington, D.C.: Lawrence Erlbaum Associates, Inc.

Zwick, R. 2006. Higher Education admissions testing. In Educational Measurement (4th ed.), edited by R. L. Brennan. Westport, CT: American Council on Education/Praeger.

Related Posts

See all Posts