By Joe Tate
A list of the most frequently studied student success levers available to leaders in higher education might include strategic advising interventions, engaging curriculum, faculty development, financial support programs, or workshops and tutoring resources—all of which would be sensible inclusions. Academic policy may not rank high (or at all) on some lists, but a discerning leader seeks hidden gems of innovation in every facet of the organization. Consider the rationale behind any given academic policy; an institution may, for example, establish a minimum grade requirement to ensure student mastery of a critical concept, or a required remediation process to help a struggling student make up ground toward that level of mastery. Beyond the obvious proximate purposes served by each of those examples is a nearly universal ultimate purpose, which is to promote student success—an admittedly broad term that is used so frequently perhaps because it succinctly encapsulates the range of unique aspirations of each student, and the various institutional statistics used to measure progress toward them. If college and university administrators accept that premise to be true, their stewardship of academic policies ought to be guided by the recognition that academic policy is a legitimate and worthwhile focus for innovation, and by a corresponding need to know whether the institution’s academic policies are, in fact, contributing to positive outcomes in relevant measures of student success.
Monitoring impact on key performance measures seems to be a natural phase in the policy change processes of organizations across other industries. Healthcare organizations closely analyze the impact of patient care policy changes on patient outcomes (Kim, et al. 2021). Major retailers similarly evaluate the impact of return and exchange policy adjustments on their success metrics (Karlsson, et al. 2023). If higher education administrators determine the need for a change in academic policy, do they likewise analyze the impact of that policy change on institutional measures of student success?
The answer to that question might amount to a matter of resources for many institutions, especially if it is assumed that such analysis should look like a formal research study. Commissioning data scientists to analyze the impact of policy changes and initiatives can yield valuable insights, which is evident in published research evaluating the impact of experimental strategies such as CUNY’s Accelerated Study in Associate Programs initiative (Azurdia and Galkin 2020). However, that level of analysis can be time-consuming, labor-intensive, and expensive. Higher education administration operates in a complex environment, characterized by continuous refinement of programs, courses, curriculum, systems, and strategies to fulfil the institution’s mission—all of which can serve as catalysts for academic policy updates. Given the level and frequency of such change in higher education, a methodologically orthodox analysis of every student-success-oriented policy update as an operational regularity is not realistic—and, more importantly, is not necessary. Researchers from prominent universities recently collaborated with business leaders to publish a Harvard Business Review article highlighting the value of “democratizing” experimentation (i.e., the analytical methods organizations use to assess the impact of new initiatives, products, services, or policies). They observed that “scaling up experimentation entails moving away from a data-scientist-centric approach to one that empowers everyone… to run experiments.” (Bojinov, et al. 2025, 99).
The democratized experimentation mindset has been adopted by many prominent organizations, including Amazon, Meta, Microsoft, and Netflix (Bojinov, et al. 2025). The rapid growth of the online travel company Booking.com has been attributed to its investment in creating a culture of experimentation, in which employees are empowered and encouraged to conduct experiments to test the efficacy of different customer service approaches (Thomke 2022). Organizational transformation expert Brian Klapper (2013, 96) has written about coaching organizations that adopt a similar mindset, in which successful change is pursued through “traditional problem solving, in the form of the scientific method.”
Academic policies, however, differ significantly from patient care and refund and exchange policies. Academic policies preserve the integrity of the educational process, provide a framework that ensures sufficient rigor to produce appropriate learning outcomes, and establish expectations to help students navigate academia in their pursuit of knowledge and associated credentials. Higher education administrators may therefore be disinclined to consider academic policy to be as malleable as a merchandise refund policy or the style guide standards for a travel website—and rightly so. Academic policy’s unique purpose in higher education deserves due deference but should not render it off-limits to innovation. One of the primary roles of higher education has been to promote and advance the acquisition of knowledge. Evaluating the student success merits of academic policies can be considered an exercise in turning that acquisitional prowess inward.
Klapper (2013, 96) suggests that the pursuit of knowledge for organizational transformation can be distilled down to what is taught in grade school classrooms, noting that the scientific method “challenges you to make an observation, ask a question, form a hypothesis, conduct an experiment, and then accept or reject your hypothesis.” However, promoting a consistent approach to this type of scientific inquiry into academic policy efficacy poses some challenges, as the problems driving academic policy changes can arise from every corner of an institution. The evaluation of an academic policy’s impact on student success requires an institutional champion to lead that effort and to promote a consistent approach. Registrars are optimally situated to be those champions in systematizing a process of observing, hypothesizing, testing (evaluating), and applying learning, based on the typical institutional administrative structure and their proximity to student data and academic policy enforcement.
Observing: Identifying the Purpose of Academic Policy Changes
The registrar’s office may not be the origin of most academic policy changes, which means its observations might not occur until after an academic policy change has already been proposed. The registrar’s role, then, is to initiate the evaluative process by assessing the significance of in-flight (or recently approved) policy changes as they pertain to student success outcomes. Policy changes can be classified based on the purpose behind each change. Some policy changes may not be suitable candidates for evaluation because they were implemented for reasons that don’t necessarily invite deeper inquiry. For example, a policy change may have been implemented only to clarify an existing policy or to meet a mandatory state, federal, or licensure requirement. In the spirit of optimizing limited resources, the registrar’s observations can be attuned to identifying policies specifically intended to influence student success outcomes and designating them for further analysis.
Hypothesizing: Articulating the Expected Outcomes of Academic Policy Changes
If a policy is intended to influence student success outcomes, its sponsors should have some idea of which outcomes they expect to be influenced (e.g., course grades, GPA, rate of progression, graduation rates). Registrars can promote uniformity and systematization by articulating a hypothesis based on those expected outcomes and informed by their anticipated future evaluation of the hypothesis. Registrar’s office guidance may also include identifying the parameters of future evaluation, including who will be in the test group and who will be in the control group (i.e., which students are impacted by the policy change, and which are not). Some policies are conducive to having a test population and control population that are contemporaries; for example, a policy change that provides students an alternative, optional course to use in satisfying a program requirement would allow for a test population (those taking the new course) and a control population (those taking the existing course or courses) that are both engaging in the observed activity at the same time. Conversely, if a policy change impacts all students in a program or degree level, the control group may need to consist of students who completed the relevant activity or programmatic milestone prior to the policy change. In either scenario, a simple, clear hypothesis based on reasonable comparisons and expected outcomes is essential for ensuring effective evaluation. Rushed or perfunctory hypothesizing can lead to skewed evaluations—for example, hypothesizing inflated outcomes for the test population, resulting in an erroneously high bar for hypothesis acceptance and yielding invalid conclusions.
Evaluating: Comparing the Outcomes of Test and Control Populations
The registrar’s office can evaluate hypotheses after sufficient time has passed by querying student data for test and control populations. Analyzing student datasets should include identifying and accounting for variables that might skew results (e.g., the influence on outcomes of classes taken in a summer term or students’ academic standing within each population), and examining obvious variations and differences in outcomes between the two populations. The registrar’s office and other policy stakeholders can then use professional judgment in weighing the significance of differences between test and control population outcomes to form conclusions about the hypothesis. Professional judgment can be supplemented with the output of generative artificial intelligence tools that have shown promise in their capacity to provide basic statistical analysis—in particular, analyzing the relative significance of differences between test and control population data using basic two-grouped comparisons like independent and paired t-tests—with reliable accuracy (Shahrul and Syed Mohamed 2024).
Applying Learning: Acting on Insights Gained from Accepted or Rejected Hypotheses
If the registrar’s office finds no significant difference between the test and control populations, the hypothesis can be rejected. A rejected hypothesis does not necessarily warrant a rollback of the policy. If the policy has other merits, its sponsors and academic policy governance may elect to leave it in place or pursue adjustments to further improve the measure of interest. The hypothesis is also rejected if the test population significantly underperformed compared to the control population. Significant underperformance might warrant a repeal of the policy if the results are unambiguous, but more likely would prompt additional review that could lead to another modification of the policy or (if the stakes are high) commissioning a deeper-dive analysis by the institution’s data science experts to further validate results. If the test population’s outcomes are significantly different in a positive direction as compared to the control population, the hypothesis is accepted—which may likewise lead to an interest in further exploration of the policy’s impact, including potentially expanding it to other student populations that could benefit (e.g., apply the policy to students in other programs, other colleges, other program levels, etc.). The results of these evaluations can be shared with policy sponsors and the institution’s academic policy governance body. They also form a valuable organizational artifact. In addition to evaluating the efficacy of one particular policy change, the results of each analysis can be archived as a resource to inform future problem-solving and innovation efforts (Bojinov, et al. 2025).
Implementing Systematic Policy Evaluation: Suggestions for Initial Steps
Registrars interested in expanding their role as policy analysts might consider the following as prudent prerequisites to demonstrating the value of, and soliciting support for, a systematic approach to policy evaluation.
Strengthen Analytical Competencies: Leverage available learning and upskilling resources to enhance registrar’s office staff capabilities in data analysis, including the judicious supplemental use of generative AI tools.
Create a Policy Change Inventory: Establish a database, spreadsheet, SharePoint list, or workflow for recording every academic policy change and classifying each based on its intended purpose (e.g., improve student success, clarify policy, meet licensure or regulatory requirements, etc.)—which can also serve as a record for policy change hypotheses and evaluation outcomes.
Confirm Data Access: Confirm the capability and secure the support of those responsible for querying student record data to be used in evaluating hypotheses.
Pilot the Process: Select one or two policy changes for pilot analysis; share results with key policy stakeholders and solicit their feedback.
Conclusion
Far from a rigid set of sacrosanct rules to be untouched over time, academic policy is a legitimate area of consideration for institutional leaders pursuing optimal student success outcomes. Integrating regular policy evaluation into institutional strategic analyses and administrative operations may require a change of mindset in which academic policy is recognized as a legitimate and sensible area of focus for innovation. The subsequent need for further analysis may require registrars to establish (or enhance) their roles as policy evaluators and as stewards best suited to systematically articulate and evaluate the hypotheses underlying academic policy changes. The documented experience of innovators across industries affirms that such an approach yields dividends—and, in higher education, those dividends take the laudable form of imparting knowledge and improving human lives.
References
Azurdia, G., and K. Galkin. 2020. An Eight-Year Cost Analysis from a Randomized Controlled Trial of CUNY’s Accelerated Study in Associate Programs. New York: MDRC.
Bojinov, I., D. Holtz, R. Johari, S. Schmit, and M. Tingley. 2025. Want your company to get better at experimentation? Harvard Business Review. 103(1): 96–103.
Karlsson, S., P. Oghazi, D. Hellstrom, P. C. Patel, C. Papadopoulou, and K. Hjort. 2023. Retail returns management strategy: An alignment perspective. Journal of Innovation and Knowledge. 8(4): 1–10.
Kim, M. P., Y. Y. Law, D. T. Nguyen, S. L. Jones, E. A. Graviss, and R. A. Phillips. 2022. Health system strategy to safely provide surgical care during the COVID-19 pandemic. NEJM Catalyst Innovations in Care Delivery. 3(2).
Klapper, B. 2013. The Q-Loop: The Art & Science of Lasting Corporate Change. Brookline, MA: Bibliomotion.
Shahrul, A. I., and A. M. F. Syed Mohamed. 2024. A comparative evaluation of statistical product and service solutions (SPSS) and ChatGPT-4 in statistical analyses. Cureus. 16(10).
About the Author
Joe Tate serves as Senior Director of Program and Policy Implementation in the Registrar’s Office and as Associate Faculty at University of Phoenix. He has eighteen years of experience in higher education administration and has published articles and presented sessions at AACRAO annual meetings on academic policy and quality management. He is an ASQ Certified Quality Auditor and a certified Project Management Professional (PMP). He holds a Master of Arts in English from Northern Arizona University and a Master of Business Administration from University of Phoenix.
share