National Collegiate Athletic Association
NCAA News Archive - 2007
« back to 2007 | Back to NCAA News Archive Index
Division I prepares to live without APR safety net
|
By Michelle Brutlag Hosick
The NCAA News
The NCAA News
Starting in spring 2008, potentially hundreds of teams that had previously avoided penalties in the academic-reform system will be forced to pay the piper when the statistical adjustment that gave them the temporary pass is eliminated.
The squad-size adjustment, essentially a statistical “confidence interval” around a team’s actual Academic Progress Rate, was developed at the beginning of the academic-reform movement to provide statistical protection for teams with smaller squads.
The Division I Committee on Academic Performance, the group charged with implementing and monitoring academic reform, realized the difficulty in predicting the long-term behavior of a team with a high level of statistical confidence when the amount of data is small. For example, a golf team that awards only five scholarships per year, or even a basketball team that awards the full complement of grants-in-aid, would have been subject to wild variations in APR based on the outcomes of just one or two student-athletes.
The CAP announced it was applying the squad-size adjustment for the first three years of APR data collection until teams were able to build up enough data points to provide an accurate depiction of academic behavior. The amount of data in one year is just too small to be a reliable portrayal.
However, once the fourth year of APR data is collected — in 2007-08 — enough data will have been collected to make the adjustment unnecessary in most cases.
That means smaller teams that have become accustomed to receiving a “get out of jail free” card because of the squad-size adjustment are running short on time to make a behavior adjustment.
The 30 threshold
The academic activity occurring in teams right now (for which data will be collected during the 2007-08 academic year) will be the first to be evaluated without the adjustment — for example, basketball teams’ current academic performance will be judged on face value, without the use of a “confidence interval.”
Without the adjustment, a large number of teams would have been adversely affected by a single academic casualty simply because of the small number of people included in the cohort.
“One person could make a big difference in a small team, so we used something that’s pretty standard in statistics called the confidence interval,” said Jack McArdle, senior professor at the University of Southern California and research consultant to the NCAA. “When a sample size is small, the confidence boundary around it is very large to recognize that one person could make a difference … It really affects only teams that have fewer than 30 members.”
When one person can impact the APR in a negative way for reasons other than academic casualty (such as transfer for personal reasons or to pursue a course of study not offered at the first institution), the rate becomes watered down and does not reflect what it is intended to — which is to determine whether an entire team is performing well academically.
Jack Evans, a member of the Division I CAP and faculty athletics representative at the University of North Carolina, Chapel Hill, said with such a small sample, a low rate “might not have anything to do with whether a particular group was paying appropriate attention to its academic responsibilities.”
Thus, the squad-size adjustment wasn’t a way to let teams off the hook, but rather a statistically sound application to protect the integrity of reform.
“The idea behind the squad-size adjustment was to reduce the chance of assigning a penalty when the penalty wasn’t deserved,” Evans said.
Why then, will it be eliminated beginning with the 2007-08 data collection?
Most teams will have accumulated enough data to reach the threshold at which a solid pattern starts to emerge. McArdle said a set of data that includes 30 student-athletes is standard. Most teams have the opportunity over the course of four years to build up a data set that includes 30 student-athletes, with each student athlete counting once for each year he or she is in the cohort. For example, a student-athlete who participates on an athletics team and receives athletics aid for four years counts four times.
Keeping the squad-size adjustment for that length of time also allowed time for actual behavioral change among all teams.
How many will be affected?
With only two years of data currently available, it is difficult to predict exactly how many teams will be caught in the penalty net when the squad-size adjustment goes away after the completion of four years of data is released in spring 2008 (the third year of data will be released this spring).
Using the data from 2003-04 and 2004-05, the number of teams penalized could increase between four and five times once the squad-size adjustment is eliminated. Only 3.5 percent of teams posted two-year APRs below 925 with the squad-size adjustment in place. Without the adjustment, 15.4 percent were below the mark required to avoid contemporaneous penalty.
Projections are that baseball, men’s basketball and football will see the largest increases in teams penalized after the elimination of the squad-size adjustment. For example, of 326 total men’s basketball squads, 100 had an APR below 925 but did not face contemporaneous penalties because of the squad-size adjustment. An additional 18 teams did not face a penalty for other reasons (including institutional characteristics or other waivers), and 19 teams were penalized.
Two-year data show that most teams are not impacted at all by the squad-size adjustment — nearly 85 percent of all teams using the metric have a raw APR at 925 or above.
Though the squad-size adjustment is going away, it will be retained in a small number of instances. If some teams do not have the requisite number of student-athletes in their cohort (30) over the four-year data-collection period, those teams will still be entitled to use the squad-size adjustment.
“We’re still going to give them the benefit of the doubt,” Evans said. “In fact, that’s really what the squad-size adjustment is about — it’s giving a team the benefit of the doubt, reducing the risk of assigning a penalty in a situation when it’s not deserved.”
Evans said coaches should not be surprised when a rate that doesn’t warrant a penalty this year does next year.
“I hope there are not a lot of situations in which people are going to say, ‘Oh, you caught me by surprise,’ because we’ve been talking about this from the beginning,” Evans said. “I recognize the real possibility that some coach might just say, ‘The only thing that matters to me is whether I get a penalty or not, don’t bother me about these fine points.’ A coach who has those sort of blinders on might feel surprised, even though we have a right to feel that he shouldn’t be.”
With the way the APR is calculated, a coach can determine exactly what needs to happen — or what can’t be afforded — to avoid penalty for his or her squad. McArdle said it might take some “sophisticated counting,” but figuring out a team’s four-year APR is relatively simple.
“They could (then) try all the scenarios for everybody and they could know what they’ll need to guard against (to avoid penalties),” he said.
The NCAA membership services staff is working to ensure that no coach has those blinders on and that every institution is aware that the confidence boundary will be eliminated for the next cycle. To that end, staff members are emphasizing the change at meetings with athletics conferences, academic groups, coaches associations and regional seminars. A warning will also be included on the institution’s APR report generated by the NCAA this spring.
The squad-size adjustment, essentially a statistical “confidence interval” around a team’s actual Academic Progress Rate, was developed at the beginning of the academic-reform movement to provide statistical protection for teams with smaller squads.
The Division I Committee on Academic Performance, the group charged with implementing and monitoring academic reform, realized the difficulty in predicting the long-term behavior of a team with a high level of statistical confidence when the amount of data is small. For example, a golf team that awards only five scholarships per year, or even a basketball team that awards the full complement of grants-in-aid, would have been subject to wild variations in APR based on the outcomes of just one or two student-athletes.
The CAP announced it was applying the squad-size adjustment for the first three years of APR data collection until teams were able to build up enough data points to provide an accurate depiction of academic behavior. The amount of data in one year is just too small to be a reliable portrayal.
However, once the fourth year of APR data is collected — in 2007-08 — enough data will have been collected to make the adjustment unnecessary in most cases.
That means smaller teams that have become accustomed to receiving a “get out of jail free” card because of the squad-size adjustment are running short on time to make a behavior adjustment.
The 30 threshold
The academic activity occurring in teams right now (for which data will be collected during the 2007-08 academic year) will be the first to be evaluated without the adjustment — for example, basketball teams’ current academic performance will be judged on face value, without the use of a “confidence interval.”
Without the adjustment, a large number of teams would have been adversely affected by a single academic casualty simply because of the small number of people included in the cohort.
“One person could make a big difference in a small team, so we used something that’s pretty standard in statistics called the confidence interval,” said Jack McArdle, senior professor at the University of Southern California and research consultant to the NCAA. “When a sample size is small, the confidence boundary around it is very large to recognize that one person could make a difference … It really affects only teams that have fewer than 30 members.”
When one person can impact the APR in a negative way for reasons other than academic casualty (such as transfer for personal reasons or to pursue a course of study not offered at the first institution), the rate becomes watered down and does not reflect what it is intended to — which is to determine whether an entire team is performing well academically.
Jack Evans, a member of the Division I CAP and faculty athletics representative at the University of North Carolina, Chapel Hill, said with such a small sample, a low rate “might not have anything to do with whether a particular group was paying appropriate attention to its academic responsibilities.”Thus, the squad-size adjustment wasn’t a way to let teams off the hook, but rather a statistically sound application to protect the integrity of reform.
“The idea behind the squad-size adjustment was to reduce the chance of assigning a penalty when the penalty wasn’t deserved,” Evans said.
Why then, will it be eliminated beginning with the 2007-08 data collection?
Most teams will have accumulated enough data to reach the threshold at which a solid pattern starts to emerge. McArdle said a set of data that includes 30 student-athletes is standard. Most teams have the opportunity over the course of four years to build up a data set that includes 30 student-athletes, with each student athlete counting once for each year he or she is in the cohort. For example, a student-athlete who participates on an athletics team and receives athletics aid for four years counts four times.
Keeping the squad-size adjustment for that length of time also allowed time for actual behavioral change among all teams.
How many will be affected?
With only two years of data currently available, it is difficult to predict exactly how many teams will be caught in the penalty net when the squad-size adjustment goes away after the completion of four years of data is released in spring 2008 (the third year of data will be released this spring).
Using the data from 2003-04 and 2004-05, the number of teams penalized could increase between four and five times once the squad-size adjustment is eliminated. Only 3.5 percent of teams posted two-year APRs below 925 with the squad-size adjustment in place. Without the adjustment, 15.4 percent were below the mark required to avoid contemporaneous penalty.
Projections are that baseball, men’s basketball and football will see the largest increases in teams penalized after the elimination of the squad-size adjustment. For example, of 326 total men’s basketball squads, 100 had an APR below 925 but did not face contemporaneous penalties because of the squad-size adjustment. An additional 18 teams did not face a penalty for other reasons (including institutional characteristics or other waivers), and 19 teams were penalized.
Two-year data show that most teams are not impacted at all by the squad-size adjustment — nearly 85 percent of all teams using the metric have a raw APR at 925 or above.
Though the squad-size adjustment is going away, it will be retained in a small number of instances. If some teams do not have the requisite number of student-athletes in their cohort (30) over the four-year data-collection period, those teams will still be entitled to use the squad-size adjustment.
“We’re still going to give them the benefit of the doubt,” Evans said. “In fact, that’s really what the squad-size adjustment is about — it’s giving a team the benefit of the doubt, reducing the risk of assigning a penalty in a situation when it’s not deserved.”
Evans said coaches should not be surprised when a rate that doesn’t warrant a penalty this year does next year.
“I hope there are not a lot of situations in which people are going to say, ‘Oh, you caught me by surprise,’ because we’ve been talking about this from the beginning,” Evans said. “I recognize the real possibility that some coach might just say, ‘The only thing that matters to me is whether I get a penalty or not, don’t bother me about these fine points.’ A coach who has those sort of blinders on might feel surprised, even though we have a right to feel that he shouldn’t be.”
With the way the APR is calculated, a coach can determine exactly what needs to happen — or what can’t be afforded — to avoid penalty for his or her squad. McArdle said it might take some “sophisticated counting,” but figuring out a team’s four-year APR is relatively simple.
“They could (then) try all the scenarios for everybody and they could know what they’ll need to guard against (to avoid penalties),” he said.
The NCAA membership services staff is working to ensure that no coach has those blinders on and that every institution is aware that the confidence boundary will be eliminated for the next cycle. To that end, staff members are emphasizing the change at meetings with athletics conferences, academic groups, coaches associations and regional seminars. A warning will also be included on the institution’s APR report generated by the NCAA this spring.
© 2010 The National Collegiate Athletic Association
Terms and Conditions | Privacy Policy