The Chronicle of Higher Education

From the issue dated November 17, 2000

Are Students Actually Learning?

Survey offers way to assess undergraduates' satisfaction -- and possibly to evaluate colleges

By LEO REISBERG< P> While guiding prospective students and their parents on a tour of Elon College last month, Michael Garcia was greeted by a familiar faculty member.

"Hi, Mike," the woman said, "how's your knee doing?"

"Oh, is that your professor?" a parent asked Mr. Garcia, a senior majoring

ALSO SEE:

Highlights From the National Survey of Student Engagement

Exemplary Institutions

With the Aim of Retaining Freshmen, a Survey Examines Their Experience

Researcher Examines Student 'Outcomes'
in broadcast communications. "Yeah, I had her freshman year," he responded, still amazed that she remembered his knee surgery three years ago.

It was certainly a convenient encounter, but one that Mr. Garcia says is typical at Elon, a comprehensive North Carolina college of fewer than 4,000 undergraduates.

The size of the campus facilitates close interactions among students and faculty members, he says. Professors encourage students to call them at home to discuss class assignments, grades, and life in general. In the classroom, active participation is more common than droning lectures.

If all this sounds like rhetoric one might expect to hear from a loyal student leader, a highly anticipated national study suggests he is not just blowing smoke. What some are calling a potential alternative to the college rankings, the National Survey of Student Engagement purports to be a new approach to measuring institutional quality.

Elon and three liberal-arts colleges -- Beloit, Centre, and Sweet Briar -- stood out among 276 institutions whose freshmen and seniors participated. The survey, a $3.3-million project sponsored by the Pew

Exemplary Institutions

Following is an explanation offered by the researchers who prepared the National Survey of Student Engagement on identifying potentially exemplary institutions:

There are different ways to think and talk about educational effectiveness. One way is to determine those institutions at which students perform at a very high level on one or more benchmarks. With this information in hand, other colleges and universities aspiring to engage their students at high levels could turn to these institutions for ideas, especially those with which they have certain features in common, such as size, educational mission, student characteristics, and other factors.

We illustrate how this could be done using NSSE 2000 results for one benchmark level of academic challenge. In the absence of data spanning multiple years, we are not at this point declaring that these schools are necessarily exemplars to be emulated. At the same time, their students reported being highly engaged in Spring 2000 in these areas.

STRONG PERFORMERS: Academic Challenge

FIRST-YEAR STUDENTS

Liberal-Arts Colleges
Antioch College
Centre College
Denison University
Wabash College

General Colleges
Columbia College (S.C.)
Marymount College (N.Y.)
Medgar Evers College of the City University of New York

Master's Institutions
College of Notre Dame of Maryland
Loyola College in Maryland
Regis College (Mass.)
University of Richmond

Doctoral-Intensive Universities
Pepperdine University
Polytechnic University (N.Y.)
Seton Hall University

Doctoral-Extensive Universities
American University
Indiana University at Bloomington
Rice University
University of Michigan at Ann Arbor

Special Mission
Rhode Island School of Design
Rose-Hulman Institute of Technology


SENIORS

Liberal-Arts Colleges
Antioch College
Centre College
Evergreen State College
Sweet Briar College
Wesleyan College (Ga.)

General Colleges
Barton College
Columbia College (S.C.)
Covenant College

Master's Institutions
College of Saint Catherine (Minn.)
Regis College (Mass.)
Saint Michael's College
University of Richmond

Doctoral-Intensive Universities
Miami University
Pepperdine University
State University of New York College of Environmental Science and Forestry

Doctoral-Extensive Universities
Brigham Young University
Loyola University of Chicago
University of Michigan at Ann Arbor
University of Virginia

Note: The institutions are listed in alphabetical order and scored high on the respective benchmark for schools of their Carnegie 2000 type. A different set of institutions might result if institutional scores were adjusted based on institutional selectivity, resources, and other variables.

Source: National Survey of Student Engagement, Indiana University
Charitable Trusts, gauges the extent to which colleges encourage actual learning, by scoring students' responses to 40 questions. More than 63,000 undergraduates filled out the questionnaire, called the College Student Report, last spring.

The questions were clustered into five "national benchmarks of effective educational practices": level of academic challenge; the amount of "active and collaborative learning," which would include how often students made class presentations, worked on group and community projects, and tutored others; student interaction with faculty members; access to enriching educational experiences like internships and study-abroad programs; and level of campus support, gauged by factors like how much the college helps students cope with non-academic responsibilities and supports social life.

The study is one of three being developed by some of the biggest names in higher-education assessment that attempt to give consumers and colleges information that may be more useful than the rankings. The other surveys, which measure the success of freshmen and alumni, focus on many of the same institutions, and the developers of all three are looking at ways to coordinate their efforts.

In a foreword to the report of the national engagement survey, released this week, Russell Edgerton, director of the Pew Forum on Undergraduate Learning, and Lee Shulman, president of the Carnegie Foundation for the Advancement of Teaching, said that the report "marks the public unveiling of a new diagnostic tool." Pew and Carnegie sponsored the study and helped interpret the results.

"As we all know, students can be surrounded by impressive resources and yet rarely encounter classes or other activities that authentically engage them in learning," they wrote. "The N.S.S.E. 2000 Report reveals whether and how institutions are actually using their resources to provide deep, meaningful learning experiences as reported by the students themselves."

To encourage participation, survey organizers agreed to keep institutional data confidential, leaving it up to the colleges to decide whether they wanted to make their results public. The colleges that are identified in the report as standouts in each category granted permission to be named.

The report's authors found the national results both promising and disappointing.

On the whole, students at small colleges and liberal-arts colleges came out ahead in the five benchmark categories, according to the national report. Beloit, Centre, Elon, and Sweet Briar were the only four colleges that scored in the top 20 percent in all five benchmark categories for both freshmen and seniors.

"There are some really hidden jewels out there," says George D. Kuh, a higher-education professor at Indiana University at Bloomington and director of the survey. "One surprise was that some high-profile schools don't do as well as some schools that most people have never heard of."

The National Survey of Student Engagement found that liberal-arts colleges were more demanding academically, requiring more writing and more analytical thinking than other types of colleges.

But the scores varied widely within each category of institutions. Some small liberal-arts colleges did poorly in some categories, while some large, doctorate-granting universities did well.

Most students surveyed (79 percent) said that their institution expects them to study a significant amount. But less than 15 percent of full-time freshmen and seniors alike spend 26 or more hours a week studying (the traditional standard, the report says, is about 30 hours a week for full-time students). About half of the students said they spend between six and 15 hours a week, and one in 10 full-time students spends five or fewer hours a week preparing for class.

The survey also found that students at large universities are the least involved in activities that promote collaborative learning. "Arguably, these are the very institutions where active and collaborative learning approaches are most needed to compensate for the anonymity and passivity that can characterize large, impersonal learning environments," the report says.

Sixty-three percent of seniors who were surveyed participated in community service, and 41 percent were involved in a community-based project as part of a regular course. But 19 percent of first-year students never made a class presentation.

Across the board, the benchmark that most colleges struggled with was student interaction with faculty members. It was most common at liberal-arts colleges and least common at doctorate-granting institutions, where 53 percent of freshmen and 35 percent of seniors "never" discussed ideas from their readings or courses with a faculty member outside the classroom, and 79 percent of freshmen and 63 percent of seniors never worked with a faculty member in a non-academic venue.

The study also found that at the doctoral-intensive universities -- which, "given their exceptional faculty resources," the authors wrote, should have been expected to "lead the pack" -- only 17 percent of freshmen reported working with their professors on research, the lowest of any college category. At least one college that did poorly in the student-faculty-interaction category is already taking steps to improve.

The Georgia Institute of Technology has committed $250,000 to help faculty members involve more undergraduates in research projects. For example, some money might be used to send student presenters to research conferences.

Georgia Tech's score for both freshmen and seniors indicated that, on average, students work on research projects with faculty members somewhere between "never" and "occasionally," says Robert C. McMath, vice provost for undergraduate studies and academic affairs.

"The value for us in this survey was that it confirms things that we already know intuitively," he says. "At a big research university like ours, undergraduate students may have only limited contact with professors outside of class and there may be at least a perception that senior researchers are not much engaged with undergraduates."

Colleges that did well in the survey credited specific programs that promote student engagement.

Students in Elon College's general-studies program, for example, are required to participate in at least one of five "Elon Experiences": study abroad, internship, community service, leadership development, and undergraduate research. Of the students who graduated this year, half had studied abroad, three quarters did at least one internship, and two-thirds had participated in a community-service program, officials said.

In 1994, the college restructured its curriculum, adding a fourth hour to each three-credit course and dedicating that hour to "active learning" in every major. A 36-hour program in political science, for instance, would include nine four-hour courses instead of 12 three-hour courses. "It's not another hour of lecture," explains Gerald L. Francis, Elon's provost. ''Part of this would be time spent in class where students would have to work together collaboratively."

Julia Treu-Fowler, a sophomore at Beloit, was not surprised that the college fared so well. In the course of one weekend this month, she says, the college illustrated how it encourages a host of "effective educational practices" that are outlined in the survey: facilitates collaborative learning

Highlights From the National Survey of Student Engagement

Campus environment emphasizes spending significant amounts of time studying and on academic work
First-year
students
Seniors
Very little 2.8% 3.1%
Some 17.5% 18.2%
Quite a bit 42.9% 41.9%
Very much 36.8% 36.8%

Number of written papers or reports of 20 pages or more
First-year
students
Seniors
None 83.0% 47.2%
Fewer than 5 13.4% 42.7%
5 to 10 2.0% 7.4%
11 to 20 0.7% 1.8%
More than 20 0.9% 1.0%

Number of hours per week spent preparing for class (studying, reading, writing, rehearsing, and other activities related to your academic program)
First-year
students
Seniors
5 or fewer 9.8% 10.8%
6 to 10 24.8% 25.2%
11 to 15 22.8% 21.1%
16 to 20 18.7% 17.2%
21 to 25 11.8% 11.2%
26 to 30 7.1% 7.3%
More than 30 5.0% 7.2%

Discussed ideas from your reading or classes with faculty members outside of class
First-year
students
Seniors
Never 45.5% 28.8%
Occasionally 41.1% 49.6%
Often 10.4% 15.9%
Very often 3.0% 5.8%

Worked with faculty members on activities other than course work (committees, orientation, student-life activities, etc.)
First-year
students
Seniors
Never 70.3% 55.6%
Occasionally 21.5% 28.1%
Often 5.7% 10.6%
Very often 2.5% 5.6%

Had serious conversation with students with religious beliefs, political opinions, or personal values very different from yours
First-year
students
Seniors
Never 16.2% 14.4%
Occasionally 36.7% 40.5%
Often 27.0% 26.8%
Very often 20.1% 18.3%

Had serious conversations with students of a different race or ethnicity than your own
First-year
students
Seniors
Never 17.0% 14.9%
Occasionally 35.7% 39.3%
Often 24.7% 24.8%
Very often 22.6% 21.0%

Note: The figures are based on a survey conducted in the spring of 2000 of 151,905 randomly selected first-year and senior students from 276 participating four-year public and private institutions. The response rate was 42 percent. Students were asked to report on activities during the current year.
Source: National Survey of Student Engagement, Indiana University

and student-faculty interaction, provides a supportive campus environment, and promotes discussions among students of different backgrounds.

She was one of about 150 sophomores who participated in an annual retreat at nearby Lake Geneva, part of a two-year support program that helps first-year students adjust to college life and helps sophomores prepare for the years ahead.

At the retreat, the sophomores, their faculty advisers, and staff members from the study-abroad and career-services offices brainstormed about choosing a major, finding an internship, managing stress, and avoiding the "sophomore slump." Each student also joined in a "hot topic" session, where participants broke up into groups to discuss solutions to campus problems ranging from tasteless dining-hall food and inadequate parking to excessive drinking and sexual assault.

Ms. Treu-Fowler chose the session on campus diversity and racial issues, where about 20 white, black, Latino, and Asian students talked about ways the campus could recruit more minorities, particularly African-Americans. One solution offered by the group was to help the admissions office by sending several students of various races and ethnicities to recruit minority students in the high schools.

"I'm white and I came from a small farming community," says Ms. Treu-Fowler, of Pullman, Wash., who is majoring in Spanish and international relations. "I have friends here from Argentina, Senegal, Turkey. That's great."

Although the survey organizers say that colleges like Elon and Beloit may potentially be identified as "engaging colleges," they are reluctant to hold them up as the standard-bearers just yet.

In the coming years, the people behind the engagement study hope that consistent high performers might emerge as models for other campuses, "but it's too premature to make that judgment now," the report says. "In addition, it would be inappropriate and unfortunate if identifying exemplars so that others can learn from them becomes the basis for another set of 'college rankings.'"

Mr. Edgerton says that the immediate goal of the survey is to allow colleges to identify their strengths and weaknesses as their students perceive them. Down the road, when colleges across the board decide to make the survey results public, he says, it could be used as an accountability tool for accreditors, policymakers, and news media. Perhaps even U.S. News & World Report will factor the survey results into its rankings, he says.

"The National Survey on Student Engagement reveals a portrait of a college that a survey of resources and reputation doesn't," Mr. Edgerton says.

Editors at U.S. News familiar with the engagement-survey concept say they are eager to see the report but doubt it could be incorporated into the rankings. "To have student input would be very valuable to us," says Anne McGrath, managing editor of U.S. News' America's Best Colleges. "But it's our understanding that the survey is limited in the number of schools it reaches and the results are not going to be made public across the board. At this point in time, it doesn't seem that there's a way we could immediately use what they're doing, other than as a potential source of strong story ideas."

As maligned as the rankings are, they are closely followed by college administrators who freely publicize the results when they do well and by high-school students in the middle of their college search. But college officials say that the National Survey of Student Engagement is more practical.

"It's nice if you're in the top 50, but I don't know what that means," says William J. Flanagan, vice president of student affairs and dean of students at Beloit College, which was ranked 47th among liberal-arts colleges in the U.S. News rankings this year. "It's more important to me that our students feel very strongly about their experiences here, and it suggests we're doing things that students really value."

Mr. Kuh, the survey's director, says there is a place for both.

"The N.S.S.E. isn't designed to be an alternative to U.S. News," he says. "It would be fascinating to put our data alongside U.S. News's and say: Given this school's reputation and how much it costs, are students engaged there? Am I getting my money's worth?"


Copyright © 2000
The Chronicle of Higher Education
November 17, 2000
Section: Students
Page: A67 The Chronicle of Higher Education