June 2005

 

BOARS’ Chair Sets the Record Straight on the National Merit Scholarship Program

 

The Board of Admissions and Relations with Schools (BOARS) conducts ongoing reviews of admissions criteria to ensure consistency with the University of California freshmen admissions policies and guidelines. As part of this process, BOARS recently reviewed the National Merit Scholarship Program (NMSP) and issued a recommendation that campuses reassess the use of the National Merit Scholar designation in admissions decisions and that the appropriate Senate agencies and senior management officials reconsider UC’s participation in the scholarship program.

 

A letter from former UC Associate President Patrick Hayashi, who served as a trustee of the College Board from 2000 to 2004, prompted BOARS’ review of the National Merit Scholarship Program (NMSP). In a letter dated October 27, 2004, Dr. Hayashi wrote to his fellow College Board trustees and raised several important questions about the educational soundness and fairness of the NMSP, questions initially raised by him four years previously that had remained unanswered.

 

BOARS’ investigation into the NMSP began in October 2004 and continued for several months and included a series of communications between the College Board and the National Merit Scholarship Corporation. As a result of this review, BOARS identified compelling validity and adverse impact concerns regarding the National Merit Scholarship selection procedures and concluded that extra weight in admissions decisions should not be given simply because a student is designated a National Merit Scholar. The committee has requested that all UC campuses reconsider any admissions preferences that they may be giving to National Merit Scholars. BOARS also requested the assistance of the Academic Senate Chair in asking the appropriate Senate agencies and administrative officials to evaluate the appropriateness of UC’s participation in the NMSP with respect to scholarship and financial aid policies.

 

BOARS’ recommendations regarding the National Merit Scholarship Program have received support, including a letter from the University Committee on Planning and Budget. Yet, some faculty may wish to better understand BOARS’ decision. In response, this article attempts to address some of these questions and clarify misconceptions about the reasoning behind the committee’s actions.

 

• How are National Merit Scholars selected?

 

Each October, over 2.5 million high school students take the combined Preliminary SAT/National Merit Scholarship Qualifying Test (PSAT/NMSQT). The PSAT is only offered once a year and its intended purpose is to provide 10th and 11th grade students an opportunity to practice for the SAT. The PSAT/NMSQT has three sections, two of which assess verbal skills, one of which assesses math skills.

 

The National Merit Scholarship Program uses the PSAT to establish the pool of potential scholarship winners. Simple cut-off scores are used twice to identify the pool of students from which National Merit Scholars are selected. First, a cut-off score is used to identify the top 3% of PSAT takers; these are deemed “Commended Students.” Second, the NMSP then sets a different cut-off score for each state. Thus, what appears to be one national program is in reality 50 different state programs, each with its own standard of “merit.” Students who are above these cut-off scores are given further consideration and called “Semifinalists.” Approximately 50% of “Commended Students” are eliminated at this step. Over 90% of the students who reach the semi-finals are deemed “Finalists” on the basis of a review of their applications, high school records, high school principal recommendations, and essays. Thus, the use of simple cut-off scores at the “Commended Students” and “Semi-Finalists” phases are, by far, the primary factors in determining who will win a National Merit Scholarship.

 

• What is wrong with the National Merit Scholarship Program selection procedure?

 

BOARS has identified three major concerns with the procedure employed by the NMSP to select its scholarship winners:  

• The PSAT has never been validated for use to select “meritorious” students in the manner employed by the NMSP. The College Board has argued that the PSAT is correlated with the SAT, but the PSAT is a different test and administered under entirely different conditions. The PSAT is much shorter and easier than the SAT and contains very few test questions that discriminate among high scoring students. It is also now a very different test in that the new SAT contains a writing section. BOARS found the College Board’s explanation unpersuasive because tests cannot be validated by proxy. Furthermore, tests per se are not validated; their particular uses are. To establish the validity of the PSAT for selection of meritorious students, one would have to determine that the specific way that the NMSP employed the PSAT to select scholarship winners, particularly its use of simple cut-off scores, truly distinguished between meritorious and non-meritorious students. Such a validity study has never been conducted. If such a study were undertaken, the extant literature on standardized tests suggests that the study would reveal that the selection procedures used by the NMSP are invalid.  

• The National Merit Scholarship Program uses selection procedures that violate other fundamental principles governing responsible use of standardized tests. The NMSP uses a simple, empirically invalidated, cut-off score from a single test to eliminate students. That is all it takes into account when establishing its pool. All professional associations involved in testing say that such practices are inappropriate – major decisions should never be made on the basis of small, statistically insignificant differences in test scores or on test scores alone. PSAT cut-off scores in the Commended Students and Semi-finalists rounds are extremely important to the NMSP: the vast majority of students who are not eliminated as a result win National Merit Scholarships.  

• National Merit Scholarship Program selection procedures maximize rather than minimize negative impact on under-represented and disadvantaged students. The literature is clear that sole reliance on high-stakes, norm-referenced tests like the SAT and PSAT result in severe adverse impact: our own experience with the role of tests in admissions supports this.

 

• Why is BOARS attacking merit? What’s wrong with awarding merit-based scholarships?

 

BOARS is not attacking merit. Quite to the contrary, BOARS is affirming the appropriate definition of academic merit. BOARS has made it clear that many of the National Merit Finalists are excellent students and likely to fare well at UC and in local campus scholarship competitions. BOARS is questioning whether the NMSP is an appropriate way to identify, commend and support meritorious students.

 

It is established University policy that academic merit is best determined through review of a student’s entire record of achievements, evaluated against the backdrop of their learning opportunities. Such a definition of merit is employed by the most elite colleges and universities in the nation and is one that the University has judged to be consonant with the University’s philosophy and mission.

 

BOARS found that the criterion used by the NMSP to select scholarship winners is unsupported as a definition of merit. The committee also has concerns about the practice of awarding these students non-need stipends. The National Merit Scholarship Corporation requires that any university that sponsors a National Merit Scholar must award that student a merit stipend regardless of financial need. Because of the way these students are selected, we have reason to believe that a large portion come from affluent families and do not have financial need.

 

BOARS is not against merit-based scholarships. Indeed, for students being admitted into the University of California, it is somewhat specious to juxtapose merit- to need-based awards as if those offered the latter are not meritorious. By definition, those eligible for admission into the University MUST be in the top 12.5% of graduating high school seniors.

 

UC funds that are currently used to support the National Merit Scholarships are fungible and could be used for other purposes. For example, they could be used to support campus-based merit scholarships. By participating in the NMSP, UC campuses are allowing an outside organization to select which students will be awarded UC-funded scholarships. If UC does choose to allow an outside agency to select winners of UC-funded scholarships, then the faculty and administration must make certain that the procedures for selecting the winners are ones that UC can and should endorse.

 

• Why hasn’t anyone else raised these concerns?

 

The answer is that others have. Over ten years ago, FairTest, a watchdog organization, sued the College Board, the Educational Testing Service (ETS) and the National Merit Scholarship Program for misuse of the PSAT resulting in gender bias. That case was settled between the College Board and the US Department of Education requiring changes in the way PSAT was structured and scored. BOARS understands that the supporting documents have recently been unsealed and made available to the public. The US Department of Education had asked Professor Walter M. Haney, from Boston College’s Center for the Study of Testing, Evaluation and Educational Policy to assess the soundness of the NMSP’s selection procedure. Professor Haney reached the same conclusion as BOARS for many of the same reasons: (1) the PSAT and the SAT are very different tests and the PSAT cannot be validated by reference to the SAT; (2) the PSAT, itself, has never been validated for use in the manner employed by the NMSP; and (3) the use of a simple cut-score, on the basis of a single test, in a high stakes context is educationally and psychometrically indefensible. (See Haney to Steven Pereira, US Department of Education, February 15, 1996.)

 

This issue has never been given much public scrutiny because this program operates at the margins between high schools and universities. Students and parents do not have the resources and expertise to raise questions; neither do high schools. Some colleges and universities strive to raise their status by aggressively recruiting and enrolling merit scholars based upon their sophomore scores on the PSAT. The colleges and universities that are “successful” by such efforts have little incentive to raise questions, particularly if they are not public universities. If “unsuccessful” universities complain, they will be accused of sour grapes.

 

This may be the first time a major university that successfully enrolls large numbers of National Merit Scholars has raised critical questions.

 

 

-Michael T. Brown

with Kimberly Peterson

 

.