National Collegiate Athletic Association
NCAA News Archive - 2007
« back to 2007 | Back to NCAA News Archive Index
Ranking risks and rewards
Using APR alone to compare teams may miss context
Using APR alone to compare teams may miss context
|
By Michelle Brutlag Hosick
The NCAA News
The NCAA News
An inevitable consequence of creating a numeric tool to gauge a team’s academic performance is that someone will use the numbers to compare teams at different institutions and in different sports. The Academic Progress Rate is susceptible to such usage, and some groups already have begun using the metric to rank teams after basketball tournament and postseason bowl selections.
While those comparisons might prompt positive results, such as the publicity feeding institutions’ competitive drive to improve their showing, neither the APR nor the NCAA’s Graduation Success Rate take into account the fact that no two campuses are the same. Besides varying academic missions, institutions may have other differences, including sport sponsorship, and the demographic and educational backgrounds of their student-athletes.
The team-centered APR was not created to serve as a wide-ranging comparison tool, and it does not stand alone as a number that evaluates how a particular program is helping student-athletes obtain an education. The metric simply was intended to provide a term-by-term snapshot of an individual team’s academic progress — a real-time alternative to the previously relied upon six-year graduation rate. Nowhere in that equation is the APR stipulated as a method for ranking teams selected to participate in the postseason.
“The comparisons aren’t necessarily a bad thing,” said University of Hartford President Walter Harrison, who chairs the Division I Committee on Academic Performance. “It’s just not what any of us envisioned when we developed and implemented the system. I wouldn’t urge any president or athletics director to measure the academic success of a team by looking solely at the APR.”
The main problem, Harrison and others point out, is that an APR score on one campus might tell a different academic story about a team than the identical score for the same team in the same sport on a different campus. One team’s score could indicate academic success because it is significantly higher than the rate for the entire student body, while the same score at another school might pale to the student body rate.
Also, the same APR could represent under-performance among academically capable student-athletes on a particular team or outstanding performance among a set of students who were not expected to do so well based on indicators from their high school records.
Institutional mission
Steve McDonnell, associate athletics director for academics at Texas A&M University, College Station, said comparisons should be limited to within an institution.
“The culture and the mission of an institution can be a factor, so when you’re comparing across different schools, I don’t know how fair that assessment really is,” he said.
Former CAP member Jim Schaus, athletics director at Wichita State University, said institutional differences can affect the comparison of two teams across different sports. However, he said, the APR is still a valuable metric for what it is intended to do, which is measure eligibility and retention (the two key elements necessary for graduation) for a particular team at an institution.
“You could look at whether it’s easier to retain somebody at this institution versus another, is it easier to keep somebody eligible at this institution over another,” he said. “There could be potential variances. They are not going to be identical. But despite the fact that there may be differences between an Ivy Group school and one in another conference, the bottom line is it still has a tremendous amount of merit. Overall, it’s trying to send a message that retention and eligibility are important.”
CAP member Jack Evans, faculty athletics representative at the University of North Carolina, Chapel Hill, said while the APR wasn’t intended to be used to rank and compare teams, the ranking and comparing, particularly by mainstream media, is likely inevitable. Still, he urged caution when making such comparisons.
“Academic progress rates are not an end in themselves. They tend to be an indication of what a team is going to do with regard to graduation performance,” he said. He also said not enough APR data are available to have the comparisons carry that much weight. The 2005-06 data, the third year of the APR, is in the collection and refining process and will be released later this spring. Data collection for the current year, 2006-07, will begin this fall.
McDonnell said that he believes the APR concept is a good one, but that the media and other critics should not jump to the conclusion that it necessarily reflects the true academic performance of a specific team.
“My concern is that people in the media will just assume that it’s solely academic and not take everything else into account,” he said. “For example, if you have a situation in which a coach leaves a team, you experience a significant drop in the APR because people were transferring out for a multitude of reasons that don’t necessarily have anything to do with academics. My concern is that the data are construed by the public to be totally a reflection of academic performance — it’s not a fair reflection of academic performance.”
Change behavior, not system
Others believe that the academic-reform effort could be adversely affected if, for example, most of the field for the men’s basketball championship have low APR scores. While historically based penalties that would eliminate such teams from postseason play will eventually prevent such an outcome, teams will not reach those dire consequences until they have repeatedly failed to meet benchmarks for APR performance.
“The image is a real concern,” McDonnell said.
Harrison said he believes such a doomsday scenario could prompt different outcomes.
“I hope what you’ll start to see is teams and coaches and athletics directors making changes in their program to improve the academic performance of their athletes. That would be the positive result,” he said. “What I fear is political pressure to change the system, which would be the negative result. While I certainly think we should look at constructive suggestions for how to improve the system, there will be a few who will want to change it solely because it benefits their team. That would be a bad outcome.”
Schaus said he hopes teams change their academic performance before they get so far in the penalty process that their postseason is in danger.
“Hopefully, the penalty structure will not dig real deep because there will be changes,” he said. “That’s the whole intention; it’s not meant to be a punitive program.”
While comparing teams across the same sport might not take all facets of the APR of a particular team into account, such comparisons are more appropriate than judging a football team against a basketball team or a baseball team with a lacrosse team. Jack McArdle, a professor at the University of Southern California and consultant to the CAP, said it is not statistically sound to use the APR to compare different teams — and thus different coaches — at an institution.
“You cannot compare teams within the school because they are different sample sizes,” he said. “Statistically, you can better compare football teams that are the same size across all schools.”
Comparisons that include the APR as one of the factors involved are valid. For example, the Division I baseball community paired APR data with information on student-athlete high school academic characteristics to help identify issues within the sport that couldn’t have been done with just the APR or just the high school records alone. The data will help officials in structuring a plan to address academic concerns in baseball.
With all the comparing and contrasting going on, some athletics officials hope that positive change in the form of modified behavior occurs, especially at institutions that have teams that might not be performing as well when held up to conference or sport peers.
“As they think about the whole process from recruiting to selection to academic support to the things that a coach says about the importance of academic responsibilities, I hope there is more attention being paid to the academics in those cases where we are seeing low APR levels,” Evans said.
Harrison’s hopes are similar. He cites anecdotal evidence that positive strides already are being made.
“I see it in the way people are thinking about recruiting athletes at my school and other schools in the conference,” he said. “I hear coaches talking about how certain decisions might hurt their APR. To my way of thinking, almost everything I hear in that regard is good. They’re making good academic choices.”
While those comparisons might prompt positive results, such as the publicity feeding institutions’ competitive drive to improve their showing, neither the APR nor the NCAA’s Graduation Success Rate take into account the fact that no two campuses are the same. Besides varying academic missions, institutions may have other differences, including sport sponsorship, and the demographic and educational backgrounds of their student-athletes.
The team-centered APR was not created to serve as a wide-ranging comparison tool, and it does not stand alone as a number that evaluates how a particular program is helping student-athletes obtain an education. The metric simply was intended to provide a term-by-term snapshot of an individual team’s academic progress — a real-time alternative to the previously relied upon six-year graduation rate. Nowhere in that equation is the APR stipulated as a method for ranking teams selected to participate in the postseason.
“The comparisons aren’t necessarily a bad thing,” said University of Hartford President Walter Harrison, who chairs the Division I Committee on Academic Performance. “It’s just not what any of us envisioned when we developed and implemented the system. I wouldn’t urge any president or athletics director to measure the academic success of a team by looking solely at the APR.”
The main problem, Harrison and others point out, is that an APR score on one campus might tell a different academic story about a team than the identical score for the same team in the same sport on a different campus. One team’s score could indicate academic success because it is significantly higher than the rate for the entire student body, while the same score at another school might pale to the student body rate.
Also, the same APR could represent under-performance among academically capable student-athletes on a particular team or outstanding performance among a set of students who were not expected to do so well based on indicators from their high school records.
Institutional mission
Steve McDonnell, associate athletics director for academics at Texas A&M University, College Station, said comparisons should be limited to within an institution.
“The culture and the mission of an institution can be a factor, so when you’re comparing across different schools, I don’t know how fair that assessment really is,” he said.
Former CAP member Jim Schaus, athletics director at Wichita State University, said institutional differences can affect the comparison of two teams across different sports. However, he said, the APR is still a valuable metric for what it is intended to do, which is measure eligibility and retention (the two key elements necessary for graduation) for a particular team at an institution.
“You could look at whether it’s easier to retain somebody at this institution versus another, is it easier to keep somebody eligible at this institution over another,” he said. “There could be potential variances. They are not going to be identical. But despite the fact that there may be differences between an Ivy Group school and one in another conference, the bottom line is it still has a tremendous amount of merit. Overall, it’s trying to send a message that retention and eligibility are important.”
CAP member Jack Evans, faculty athletics representative at the University of North Carolina, Chapel Hill, said while the APR wasn’t intended to be used to rank and compare teams, the ranking and comparing, particularly by mainstream media, is likely inevitable. Still, he urged caution when making such comparisons.
“Academic progress rates are not an end in themselves. They tend to be an indication of what a team is going to do with regard to graduation performance,” he said. He also said not enough APR data are available to have the comparisons carry that much weight. The 2005-06 data, the third year of the APR, is in the collection and refining process and will be released later this spring. Data collection for the current year, 2006-07, will begin this fall.
McDonnell said that he believes the APR concept is a good one, but that the media and other critics should not jump to the conclusion that it necessarily reflects the true academic performance of a specific team.
“My concern is that people in the media will just assume that it’s solely academic and not take everything else into account,” he said. “For example, if you have a situation in which a coach leaves a team, you experience a significant drop in the APR because people were transferring out for a multitude of reasons that don’t necessarily have anything to do with academics. My concern is that the data are construed by the public to be totally a reflection of academic performance — it’s not a fair reflection of academic performance.”
Change behavior, not system
Others believe that the academic-reform effort could be adversely affected if, for example, most of the field for the men’s basketball championship have low APR scores. While historically based penalties that would eliminate such teams from postseason play will eventually prevent such an outcome, teams will not reach those dire consequences until they have repeatedly failed to meet benchmarks for APR performance.
“The image is a real concern,” McDonnell said.
Harrison said he believes such a doomsday scenario could prompt different outcomes.
“I hope what you’ll start to see is teams and coaches and athletics directors making changes in their program to improve the academic performance of their athletes. That would be the positive result,” he said. “What I fear is political pressure to change the system, which would be the negative result. While I certainly think we should look at constructive suggestions for how to improve the system, there will be a few who will want to change it solely because it benefits their team. That would be a bad outcome.”
Schaus said he hopes teams change their academic performance before they get so far in the penalty process that their postseason is in danger.
“Hopefully, the penalty structure will not dig real deep because there will be changes,” he said. “That’s the whole intention; it’s not meant to be a punitive program.”
While comparing teams across the same sport might not take all facets of the APR of a particular team into account, such comparisons are more appropriate than judging a football team against a basketball team or a baseball team with a lacrosse team. Jack McArdle, a professor at the University of Southern California and consultant to the CAP, said it is not statistically sound to use the APR to compare different teams — and thus different coaches — at an institution.
“You cannot compare teams within the school because they are different sample sizes,” he said. “Statistically, you can better compare football teams that are the same size across all schools.”
Comparisons that include the APR as one of the factors involved are valid. For example, the Division I baseball community paired APR data with information on student-athlete high school academic characteristics to help identify issues within the sport that couldn’t have been done with just the APR or just the high school records alone. The data will help officials in structuring a plan to address academic concerns in baseball.
With all the comparing and contrasting going on, some athletics officials hope that positive change in the form of modified behavior occurs, especially at institutions that have teams that might not be performing as well when held up to conference or sport peers.
“As they think about the whole process from recruiting to selection to academic support to the things that a coach says about the importance of academic responsibilities, I hope there is more attention being paid to the academics in those cases where we are seeing low APR levels,” Evans said.
Harrison’s hopes are similar. He cites anecdotal evidence that positive strides already are being made.
“I see it in the way people are thinking about recruiting athletes at my school and other schools in the conference,” he said. “I hear coaches talking about how certain decisions might hurt their APR. To my way of thinking, almost everything I hear in that regard is good. They’re making good academic choices.”
© 2010 The National Collegiate Athletic Association
Terms and Conditions | Privacy Policy