British Rowing responds to the Competition Framework article by Mike Jones
In response to Mike Jones’ and Mel Harbour’s analysis of ranking data from WEHoRR and HoRR which ran under the new Competition Framework, British Rowing has responded to specific questions put forward by the authors.
WEROW asked British Rowing: Under what criteria did Bitish Rowing design the new system and under what criteria did British Rowing exclude other ranking systems? (ELO and its derivatives, TrueSkill etc.)
British Rowing: The development of the new Competition Framework, and the criteria that guided it, started with a large-scale consultation of British Rowing members in 2014. A series of regional workshops were held across the country, led by expert volunteers and with support from British Rowing staff, with input from every level of the rowing community: clubs, rowers, competition organisers, coaches, umpires and volunteers. Everyone had the chance to have their say, and from these sessions we took three key messages:
- Firstly, the majority of members wanted to compete at an appropriate level, against rowers of similar ability and expertise.
- Secondly, they wanted a Framework that discouraged the practice of crews and individuals actively trying to avoid gaining points (and by extension, avoiding racing) so they could compete at a low level.
- Lastly, there was a consensus that the old framework, which awarded points solely to the winners of events at regattas and nothing for everyone else, wasn’t the best way to create a profile that accurately reflected a rower’s abilities.
This was reflected by the fact that 85% of women and 71% of men had just two or fewer British Rowing sweep points, while 90% of both men and women had two or fewer sculling points. That meant the overwhelming majority of British Rowing members fitted into just two categories – Novice and IM.3.
In summary, through the consultation process, the rowing community asked for a new Framework to better provide fair, close and exciting racing, for all levels of ability and experience, and while improving the old status points system was considered, it was agreed that investigating a bespoke, contemporary solution was preferable.
Both ELO and TrueSkill systems were examined and considered but, as they are designed for winner/loser scenarios, rowing needed a system that had more flexibility. Cost also had to be considered where systems, such as TrueSkill, are patented.
The other consideration for rowing is that it is fairly unique given the many permutations in which an individual competitor can compete; as an individual, a pair/double or as part of a larger crew. When investigating the ranking indices of other sports, they were either individual based (e.g. golf, cycling, tennis, badminton), based on two-person teams that competed together consistently (e.g. badminton, tennis) or based on large teams, where the team, not individuals, are recognised (e.g. football, hockey, rugby).
Any new system must also work with an incredibly diverse competition structure, events of varying sizes, a wide range of skills across competition organisers, as well as reporting results in a simple and easy to understand way.
Following this consultation phase, the decision was made to develop the new Competition Framework through a bespoke Ranking Points and a Crew Ranking Index, and this is the system we have trialled over the last year.
WEROW asked British Rowing: Once British Rowing had settled on criteria, what statistical evidence did British Rowing use to guide the development of the system? Why has British Rowing not published any internal statistical studies into the new system?
British Rowing: It was always the intention to use data from the old British Rowing Online Entry system (BROE) to analyse the accuracy of the new Framework’s algorithms. However, it quickly became apparent that data collected through the old system was not robust enough as it was only designed to record accurately the overall winner of an event, not all crew placings. As soon as this was realised, more data was collected through BROE during the test phase from late 2015 and throughout 2016 and 2017 to test and evolve the system.
Once Ranking Points were published for individual rowers during the test phase we were also then able to collect anecdotal feedback, which helped to develop the system.
Going forward, following the extension of our partnership with SAS Analytics, the global leader in analytics, we will be working with them to further analyse the system and take an evidence-based approach to evolving the Framework. The project will continue to be supported by both a volunteer and staff group, and we are open to members of the rowing community joining this, where they feel they can add value and constructive input.
WEROW asked British Rowing: What evidence does British Rowing have to show that the new system is working well enough that it can be fully implemented in April?
British Rowing: The Competition Framework and the new online entry system that underpins the Framework (BROE2) has been running in test phase for over a year.
During this test phase, over 40,000 crews have been entered across 178 competitions in BROE2 (heads and regattas). Almost all competition results were examined to see if it matched expectations, and, for the majority of cases, it did. The majority of this work has been done by a committed group of volunteers, and we are extremely grateful for the work they do on behalf of the sport.
We also received feedback from heads and regattas that have used BROE2 and their broad opinion is that new Framework has resulted in closer and fairer racing. Where crews at competitions have significantly out-performed their Crew Ranking Index, we believe that this is generally due to a lack of data. This will only improve over time and with a fully implemented Competition Framework, used by all competitions.
WEROW asked British Rowing: British Rowing has stated that it wants to implement evidence-based improvements into the system. What evidence will BR be looking for and how drastically does British Rowing expect to change the system in light of new evidence?
British Rowing: Following the renewal of our partnership with SAS Analytics, they will be providing expertise in data analytics to review in excess of 65,000 competition records captured during the pilot phase to help shape the points calculations that underpin the new Competition Framework.
Alongside data analysis, we will also continue to review the technology that supports the Competition Framework to keep up with advancements in this area. We will also continue to seek feedback from the rowing community and investigate concerns or issues raised with the system, taking a data-led approach.
It would be unfair to speculate what changes we might make to the system, as prejudging this rather defeats the object of taking a data-led, evidence-based approach. However, we are fully aware of the fundamental importance of the Competition Framework to the sport of rowing in this country, and the key role it plays.
We believe the new Framework is a significant improvement on the old one and a big step towards creating closer, fairer and more exciting racing. Any changes we make will be to further this aim, for the benefit of the whole sport.