The Chinese have a proverb for just about everything, including one that's applicable to the NCAA's efforts to collect and analyze research data for the membership.

Translated, that proverb goes something like this: "Real gold can stand the trial by fire."

Through more than 20 years of collecting statistics and providing analysis on topics ranging from academic preparation to safety to finances, the work of NCAA researchers repeatedly has been tested by fire, and proven golden.

Critics outside the Association occasionally have attacked NCAA research, especially its analysis of data addressing initial-eligibility standards. In such cases, stringent independent review validated the work.

Members of NCAA committees, displaying an appropriately healthy skepticism of data in forming recommendations, routinely challenge answers from research that sometimes seem to defy day-to-day experience -- and then find themselves persuaded by the quality of the numbers to think about things differently than before.

And those within the Association responsible for making the tough decisions -- members of the Division I Board of Directors and Divisions II and III Presidents Councils -- now rely so heavily on research in their deliberations that the Division I Board of Directors formed a task force in 2000 that spends a significant portion of its time reviewing and discussing data to ensure that presidents are fully informed about issues.

In fact, it's rare that any crucial decision -- especially one involving academic performance or health and safety, but also affecting a variety of other topics -- is made before presidents are satisfied that Association researchers fully have explored the topic.

The Division I Board of Directors Task Force is a primary driver behind most of the Association's current research efforts, says Todd Petr, NCAA managing director of research since 1998 and a member of the research staff since 1987.

"This was a group of CEOs who committed extra time to study the issues related to intercollegiate athletics," Petr said. "These CEOs demanded more research data, and were conscientious to make decisions based on those data. The needs of this group continue to drive much of the research that is undertaken for the Division I membership."

Research over time

The Association's research effort has traveled a long and winding path to gain its current importance. And a look at the program's history helps explain how research has become so influential in NCAA decision-making.

The NCAA hired its first full-time researcher in 1982, primarily to address health and safety issues. It previously had solicited data from outside sources and continued to do so, also focusing on issues such as intercollegiate athletics finances via what today remains the Association's oldest non-health/safety study -- the Revenues and Expenses of Intercollegiate Athletics reports first published in 1969.

But demand for more and better research grew, driven largely by NCAA concern about the academic preparation of student-athletes.

The adoption of 1983 Convention Proposal No. 48 -- known ever since as Prop 48 -- stands as one of the most important actions in NCAA history. The most obvious reason for that importance is that it marked the first in a series of aggressive steps by the Association to improve the preparation of recruited student-athletes for the classroom rigors of higher education, with its implementation of a minimum 2.000 grade-point average in 11 core courses and standardized test score.

However, Prop 48's adoption also prompted within the Association a widespread realization that the NCAA was pursuing significant changes -- and doing so in uncomfortably blind fashion. In essence, it also was the action that gave birth to the Association's current research commitment.

With Prop 48 came an awareness by NCAA leaders that they "had no idea of the impact," Petr said. "They realized at the time that it was a big change, and they needed to do something to understand it."

The Division I Steering Committee of the NCAA Council authorized work by an outside firm to study the impact of Prop 48 before its scheduled implementation in 1986. It also formed a panel, the NCAA Special Committee for Academic Research, to oversee the project.

Those actions led to creation of the NCAA Research Committee to assume that special committee's oversight responsibility, and initiation of what came to be known as the NCAA Academic Performance Study, building on the data collected in that initial Prop 48 research.

The Association's growing commitment to obtaining data also prompted the hiring in 1985 of Ursula Walsh, who as director of research brought to the staff experience in educational research. Two years later, the staff expanded with the selection of Petr as assistant director.

Walsh soon began assembling a team of outside researchers who could analyze trends for the Association and help decision-makers understand the impact of their actions. With Prop 48 in effect and with the 1984 and 1985 freshman classes included in the initial research nearing graduation, Walsh recruited a University of Virginia faculty member, John J. (Jack) McArdle, to apply his expertise in longitudinal research to the NCAA's review.

"I asked her, is this going to be real research, or is this going to be public policy statement research, where you're just interested in defending things," McArdle recalls. "She was mortified by the question at first, then said, 'We're doing real research; we really want to know.'

"That became a turning point in a lot of what we did. We made it clear that we were doing real research -- not something just to support existing rules, but to evaluate them honestly."

And that approach ultimately would serve Association interests well, though it would be a few years before its importance became clear.

Trial by fire

Today, NCAA research regularly is praised for its quality and credibility, and the Association's researchers are lauded for their ability to make complex concepts comprehensible and respond nimbly to requests for analysis.

"My evaluation of it is extremely high, and I think it's because it's being handled by people who are extremely reputable," said James Castaneda, professor of Spanish and the faculty athletics representative at Rice University. Castaneda was a key figure in framing recent NCAA academic reform recommendations while chair of the Division I Academics/Eligibility/Compliance Cabinet's initial-eligibility subcommittee, and then as chair of the cabinet.

"I think Jack McArdle at the University of Virginia, and Todd with all of the people he works with at the NCAA, and (University of Denver researcher) Tom Paskus -- those three -- are just impeccable in their work. We would ask them at meetings, 'Can you come up with this, this would help us a lot,' and they've gone out and worked on it and come up with it at the next meeting," he said.

But if NCAA research efforts ever were scorched by fire, the flames were generated by a 1993 attack on the researchers' credibility by an Illinois member of the U.S. House of Representatives.

Prop 48, with its minimum test-score component, drew opposition from the beginning from a range of critics who pointed to concerns that the legislation disproportionately impacted minority student-athletes. Indeed, the Association's initial research into Prop 48 effects supported those concerns.

Association leaders, however, were committed to improving academic performance by student-athletes, and firmly believed that meaningful requirements for initial eligibility were necessary.

The debate smoldered for a few years, fueled by the NCAA's adoption in 1991 of Proposal No. 16, which slated implementation of even more stringent initial-eligibility requirements.

It's certain today that Prop 16 was based at least in part on a less-than-clear understanding by decision-makers about what the research available at that time actually indicated. The 1991 proposal, which would add the sliding-scale approach to initial-eligibility requirements, inadvertently overweighted test scores relative to grade-point average.

Still, with a research team in place and generating data, the NCAA was gaining ground in its efforts to understand the impact of its legislation.

Those efforts nearly were engulfed in what finally blew up into a political firestorm fanned by Illinois Rep. Cardiss Collins, who moved beyond criticism of NCAA policies to question the reliability of research data that Association leaders still were struggling to understand in reviewing decisions. Collins alleged the data were tainted by McArdle's and three other NCAA consultants' academic relationship with a prominent psychologist who held controversial views regarding race and intelligence.

NCAA leaders responded by achieving approval of a resolution (Proposal No. 174) at the 1994 Convention that authorized formation of a special committee to review the Association's initial-eligibility standards. As part of that effort, the NCAA Research Committee assumed responsibility for soliciting independent reviews of research conducted by McArdle and others. Reviewers at Stanford University and Yale University responded with praise for the researchers' analyses and expressed confidence in the objectivity of the work.

"Having to defend the data, having to stand in front of hostile crowds and show how we did it and why -- there is an element of a turning point," Petr recalls today.

During its review, the special committee noticed the research's indication of the overweighting of test scores, and proposed revisions to Prop 16 to address that problem.

"The Prop 174 committee may be the first time we actually saw data have an impact on decisions," Petr said.

"The real turning point for us was when (the special committee) finally took a look," McArdle said. "For us, it was a gut-check; we had to make sure we had done everything right, and double-check. But there were a lot of reviews that were just the best we ever got."

'Inform by knowledge and research'

There's no question that NCAA policy-making bodies pay close attention to research data today, and that those data have an impact. And decision-makers' trust in the data isn't the only explanation for that impact; it's also true that research is being shared more widely -- and more clearly.

"The earlier work I presented much as an academic exercise: You start out with an introduction, you talk methods, you give them results, and then you discuss it in the end," McArdle said. "That's a typical presentation academically, and that's the only one I knew how to give. So those are the kinds of things those poor people were sitting through for maybe four hours, and they would wait, and then the last half hour would become very exciting."

McArdle soon learned -- aided by advice from a law professor who then chaired the Research Committee, the late John Stoepler -- to give his audiences the conclusions first, then provide the details.

"It seemed after people hear what the bottom line will be, then they are quite willing to look at the details," McArdle said. "That was a big shift, a big difference, in presentation."

When the Academics/Eligibility/Compliance Cabinet's initial-eligibility subcommittee began the work in 1997 that in part led to the academic reforms recently approved by the Division I Board of Directors, that group eagerly collected all of the relevant NCAA research it could lay its hands on, then resolved during 1998 to share that information throughout the Division I membership.

"The difference between the past and what we started to do was to inform by knowledge and by research, before too much was done that was simply arbitrary," Castaneda said.

"A decision was made, around 1997 when we started working on all these issues, that we were going to be informed by research, and we simply weren't going to do things in an arbitrary way that could be attacked," he said. "We knew that to put forth any really meaningful reform package, if there were any controversial parts to it, we had to have a sound footing for what we were doing."

Castaneda said that research changed his own thinking about eliminating cut scores from standardized testing as a component of eligibility standards.

"I had wrestled with hearing reports of what previous committees had done, when they proposed going down to 600 or eliminating the cut score altogether, and I thought that was just heresy," he said. "But as we got into the research more and more, and it showed that, for example, everybody was really worried about what would happen with the group of people who were under 600 or 700 -- the research showed us that there were fewer than one-tenth of one percent who on the sliding scale would qualify by virtue of their GPA, but who had a very, very low test score.

"So, I gradually came to be convinced there were worse things than considering elimination of cut scores."

The concern about resisting arbitrary decisions came in the context of a lawsuit filed by student-athletes in 1997 challenging the NCAA's initial-eligibility requirements. A U.S. District Court judge hearing the case, Cureton vs. NCAA, cited the arbitrary nature behind establishment of minimum standardized test scores as a component of initial-eligibility standards in finding that Prop 16 did have an unintentional disparate effect on African Americans.

"I don't know who pegged the (minimum test-score) values where they pegged them, but in the Cureton case, the judge said something like, 'Who is wise enough to say that 820 is a good score, but 819 is not good enough?' He told us our standards had been arbitrarily set, and didn't seem to be based on research or real knowledge," Castaneda said.

The U.S. Circuit Court for the Third Circuit ultimately overturned first the lower court's order suspending Prop 16, then determined that the courts had no jurisdiction to review NCAA initial-eligibility rules. But that lower court's criticism of the eligibility standards still had an affect on NCAA decision-making.

NCAA researchers always have worked to provide decision-makers with informed choices, and in recent years policy-making groups actively have solicited options from research.

"We try to help them avoid the arbitrariness of putting in a rule, and the arbitrariness of picking a particular point, like a GPA of 1.8 or 1.9 -- how do you make a choice of those two?" McArdle said. "We try to show them how they can make a choice based on existing data, and just to help them understand the problem.

"Many people would say, 'We need it to be 2.0,' and others would say, 'We don't need any rule at all,' and that's the kind of decision that's made without the data. We come in at a certain point with data like that, and then at least they are informed."

Today's path of discovery

When the Division I Board of Directors formed its task force in 2000, the time that group devoted to reviewing and discussing research indicated a real desire to understand and appropriately apply the data.

"We never had more than a half-hour in front of the presidents," Petr recalls about research presentations before establishment of the task force. "They received the data and appreciated it, but they never really had time to look at it."

But during the task force's first meeting, it spent five hours with NCAA researchers.

"They realized, if we don't have the data, we can't make the decision," Petr said.

And it's clear that at least a couple of recent decisions were heavily influenced by the data. One such decision was elimination of cut scores from initial-eligibility requirements, as NCAA decision-makers traveled essentially the same path of discovery that Castenada traveled during his subcomittee's and cabinet's review of the issue.

Data also were decisive in the recent revision of continuing-eligibility requirements. The research indicated that while high-school grades and test scores were an accurate indicator of first-year college performance and a useful indicator of the likelihood of graduation, each subsequent year of college provided a more accurate indication of degree completion -- and indicated points where student-athletes most likely fell off the pace in progressing toward graduation.

It was a significant revelation in light of a "mandate" the Division I Board of Directors gave Association committees, including Castaneda's initial-eligibility subcommittee, to find ways to maximize graduation rates while minimizing adverse impacts on minorities.

"We were able to comply with the mandate given to us essentially by enhancing graduation rates with the continuing-eligibility part, and addressing adverse impact with the initial-eligibility part of the academic reform package," said Castaneda, who added that the research was crucial in guiding the membership through discussion and debate of the issues.

"I don't think we'd have been able to do what we did without the input from the research, and without the data that the research gave us. It put us on a footing where we were doing things logically, and that was what we had needed in the past."