From the issue dated November 17, 2000

Researcher Examines Student 'Outcomes'
By ANDREW BROWNSTEIN

In this age of focus groups and market research, we can instantly gauge everything from which camera angle makes a candidate look presidential to which toy best accompanies a Happy Meal.

But what about this: How confident do you feel about your ability to explain

ALSO SEE:

Are Students Actually Learning?

With the Aim of Retaining Freshmen, a Survey Examines Their Experience


the similarities between the 1995 movie romp Clueless and Emma, the Jane Austen novel on which it is based?

That provocative question has been posed to more than 38,000 alumni of 80 colleges and universities. Perhaps even more provocative is the reason it was asked. Robert Zemsky, director of the Institute for Research on Higher Education at the University of Pennsylvania, hopes the answers to it and similar questions will help provide an alternative to the popular U.S. News & World Report rankings. Unlike the magazine, which focuses on "inputs" such as accepted students' SAT scores and class ranks, Mr. Zemsky's Collegiate Results Instrument assesses "outcomes" -- what college students know and do after graduation.

Michael McPherson, president of Macalester College, says the Emma query was "a much crunchier question than those that simply gauge student satisfaction." He has reason to be excited. His alumni scored 73 percent above the norm in the area of "information finding," one of many criteria used to measure the knowledge confidence of students who graduated between 1992 and 1994.

The questions put to alumni range from the simple -- asking about their profession and whether they "read religious works" in the past year -- to complex scenarios designed to measure preparedness after college. One question asks whether alumni could find information about a new drug for a friend with high cholesterol. Another seeks to learn whether they could investigate the causes of a morale problem at a corporation.

Mr. Zemsky hatched the idea for the C.R.I., as his assessment is known, in the early 1990's with the ambitious goal of creating a Consumer Reports for higher education. The thinking was that with college education costing as much as $120,000, consumers had a right to know what they were getting for their money.

"The idea is to decide how you want to turn out when you finish college and then find an institution whose graduates answer as you do," Mr. Zemsky says.

The results show how college alumni stack up against each other in a variety of specialized criteria like students' personal values and commitment to lifelong learning.

Though the idea for C.R.I. was formed when Mr. Zemsky received support from the Pew Charitable Trusts, which is financing two other major assessment efforts, it now is supported with roughly $2-million from the U.S. Department of Education, the Knight Higher Education Collaborative, the Sloan Foundation, and $715,000 from the institutions themselves. It began in 1998 with a pilot study of 15 institutions, which last year grew to 80, including private liberal-arts colleges like Middlebury and large research powerhouses like Michigan State University.

At Macalester, which is developing a strategic plan for its future, the data were used as a way for the college to learn more about itself. The results were mixed. Mr. McPherson was surprised to learn that 61 percent of his students went on to graduate school in five years. On the down side, Mr. McPherson said he was "not happy" with the scores of Macalester alumni in the areas that measured "quantitative competence." Graduates scored 58 percent above the national threshold in that area, less than other colleges in its comparison group, which scored 65 to 72 percent above the threshold.

Ultimately, Mr. Zemsky hopes the C.R.I. will benefit not only the institutions but the consumers looking to go to them. In the competitive world of college assessment, that means only one thing: going head to head with U.S. News.

"We are not going to replace the rankings with nothing," Mr. Zemsky says. "We've got to replace them with something better, or we're stuck."

In fact, Mr. Zemsky tried unsuccessfully to interest the magazine in his idea in 1992.

Now, Peterson's, a division of the Thomson Learning education-publishing group, which produces its own college guidebook, has picked up the exclusive rights to the C.R.I. and is using a version of it on a new Web site, http://www.bestcollegepicks.com.

The transition of the C.R.I. from an academic instrument to something "productized," in the words of Michael Brannick, Peterson's president and C.E.O., might not be a smooth one. For one, Mr. Zemsky claims the alumni data reveal the "unique signature" of an institution. But the Web site combines C.R.I. data from the 80 institutions surveyed with information in the public domain, like enrollment figures and graduation rates, to extrapolate results for roughly 1,600 institutions.

Mr. Zemsky explains that while the Web-site data might not be useful for colleges, the bank of over 33,000 alumni responses provides "a highly reliable predictive model" for consumers.

A larger problem, perhaps, is that disclosure of institutional C.R.I. data remains voluntary. In the coming months, Mr. Zemsky plans to release overall results from the survey in Change magazine. Thus far, most colleges are keeping their results internal.

The conundrum is that the rankings, whatever their failings, seek to satisfy a public hunger for accountability in higher education. Mr. Zemsky admits the C.R.I.'s failure to do so is a problem. "I think all of us face the following dilemma: Are we in the service of the institutions, who may not want to release the data, or are we in the service of the consumers, who want full public dissemination of the data?"

The answer? "I'm working on it," he said.


http://chronicle.com
The Chronicle of Higher Education
November 17, 2000 Section: Students
Page: A72