Already broken? British Rowing’s new ranking system doesn’t measure up to the data
The new British Rowing Competition Framework comes into full force on April 1. There is a growing body of evidence that the system is already broken. WEROW published two articles by Mel Harbour (1,2)setting out the mathematical arguments as to why the Personal Ranking Index (PRI) is flawed. In this article, Michael Jones appraises the new system in the light of data from the Women’s Eights Head of the River and the Head of the River Race held in March 2018.
The author, Michael Jones is studying for a PhD in Accelerator Physics at the University of Manchester whilst rowing at Agecroft Rowing Club.
In preparing the article WEROW asked questions of British Rowing and their responses can be found here.
British Rowing’s new ranking system has already prompted much discussion before its official roll out on April 1st. Until now not enough data was available to give us a clear picture about how it will affect rowing in general in the UK, but with HoRR and WEHoRR publishing the ranking data of every crew who competed it is possible to make far more concrete predictions.
Under the new system rowers get points based on how many crews they beat at head races and regattas, called their “Personal Ranking Index” (PRI). In a crew each rower’s PRI is combined to give the “Crew Ranking Index” (CRI). Boats with similar CRIs are then grouped with the intention of producing close, exciting races.
For the system to meet its stated goal crews with similar CRIs should go at a similar speed – CRI should be strongly correlated with speed.
Plotting the time crews got at WEHoRR and HoRR against their CRI we can see that, overall, faster crews tended to have a higher CRI and slower crews lower CRIs.
But for the 4 out of 5 crews with a CRI below 2000 things are far less convincing. Plotting the time vs CRI for them makes it clear how little correlation there is between CRI and speed for the vast majority of crews.
It’s good that the ranking system tells us that Brookes A, who recently beat both Blue Boats in separate matches, should be faster than Trafford B, but the more important comparison for most club rowers will be the likes of Agecroft C with a CRI of 1521 beating Vesta B, who had a CRI of 1887, by almost 2 minutes and 175 places.
So whilst 80% of crews have CRIs below 2000, a CRI below 2000 tells you very little about that crew’s speed.
It’s important to consider the effect the tide and wind changing throughout the races might have had on the correlation between CRI and time. Usefully for this analysis, at HoRR, most of the crews were set off in close to CRI order, and the wind didn’t change much over the course of the race. Crews with higher CRIs therefore set off on a more favourable tide than those with lower CRI but under similar wind conditions. So the changing tide should have increased the correlation between CRI and speed at HoRR above what is actually inherent to the system.
Looking ahead to regatta season, crews with the highest CRIs will go into “champ” events, the next bunch into a “second string” events and so on. Therefore another important test for the new system is how well it can be used to group crews of a similar speed.
Grouping WEHoRR and HoRR crews by their CRI into ~50 boat divisions shows that there is an average of ~80% overlap in the speed of crews between adjacent divisions. The huge amount of overlap means that a crew taken at random and placed into the division above them would most likely see their position within the “higher” division change by only about 13 places compared to the division they actually race in. Furthermore, at WEHoRR, there were several crews who managed to beat every crew who would be placed in a higher division than them.
We can also get an idea of the verdicts that will be seen come regatta season. Based on this data, in a 2 km eights race between two crews which have been grouped according to their CRI, the verdict is likely to be about three lengths.
So grouping crews by CRI not only results in many outliers who are far faster or slower than the rest of their group, it also splits up crews which are of similar speed.
It will be interesting to see how the new system actually plays out in the summer, but this analysis demonstrates that it suffers from many of the same problems as the old system:
- most rowers only use a small range of the ranking system,
- crews in the same tier could well have wildly different speeds, and
- crews racing at a lower status will often be much faster than crews racing at a higher status.
For a system that is regarded as confusing, cumbersome and overcomplicated, the fact it suffers from many of the same problems as the old system that it was intended to fix is pretty damning.
Thank you to the WEHoRR and HoRR committees for publishing the CRI data that made this analysis possible. Hopefully over the regatta season, as more data becomes available, it will become clear why British Rowing decided to implement this system.