ORC answers: six questions about scoring

This summer’s scoring debate has been the most engaged discussion on this blog in years.

After Færder, the DH Worlds, Sandhamn Open, and the Bermuda report, I wrote down six questions and sent them to ORC.

To their credit, they answered.

Thomas Nilsson and Andy Claughton, chairman of ORC’s International Technical Committee, prepared the response below, and I publish it in full, unedited. Whatever you think about WRS, this is how a governing body should meet criticism.

Read their answers first.

My comments come after.

Questions from Peter Gustafsson, with ORC’s Responses

These answers have been prepared jointly by Thomas Nilsson and Andy Claughton, Chairman of the ORC International Technical Committee (ITC).

Hi Peter, thank you for taking the time to crystallise your observations about Weather Routing Scoring into a concise set of questions for us to answer. We have tried to keep our answers to the point, but as you can see we have a lot to get off our chests.

In short:

WRS delivers in the 21st century what the IMS pioneers sought to do in the 1980s: polar-curve-based handicapping for offshore races. We commend it to race organisers as our best offering — they can choose to use it or not. It is transparent, in the same way as the VPP is transparent through its documentation. It offers TCFs before the race start and therefore quick post-race scoring; delays may arise from human error, not systemic problems. We are building training tools to bring WRS to race organisers, and we have a support team in place.

Background

The IMS and subsequent ORC handicap systems grew from a desire to provide handicaps that were sensitive to wind speed (TWS) and point of sailing (TWA). The certificate has always had a tabulated polar speed curve and pre-calculated handicaps for a windward-leeward course in 6, 8, 10, 12, 14, 16 and 20 knots of wind.

The persistent problem has been working out what wind the fleet actually experienced during the race. For inshore races, performance curve scoring and, more recently, the “constructed course” method based on observed wind have been used. For offshore racing, APH was used — based on a wind distribution with a little bit of everything, 6–20 knots TWS, 0–180 TWA. No offshore race has ever experienced this precise distribution of conditions; that is no surprise, since it is essentially a weather forecast made in 1980 and applied for the last 40 years.

The WRS initiative offers the opportunity to create handicaps using a better estimate of the wind conditions actually seen by the fleet, with TCFs published before the race start — a scoring process identical to using APH.

1. Have we gotten worse, despite better technology?

No — this is not a real problem. Approximately 180 races have been run using WRS this season; of these, 5 required post-race recalculation of the results. This season’s delays were not a single isolated failure: they stemmed from a combination of a flawed constructed-course calculation method and insufficient pre-race quality control. ORC recognises this as a systemic gap in the process, not an inherent flaw in the WRS concept, and is implementing pre-race validation procedures using new checking tools developed this season specifically to close it.

2. Are the scoring models too complex for their own good?

This question — “why did a boat win” — exposes the fundamental limitation of using a single, immutable TCF to score an offshore race. In a one-design championship the answer is easy: the best-sailed boat won. In a handicap race the answer is always coloured by the handicap. In a windy race a Class 40 will beat a Farr 40, but in light airs it’s the reverse. No amount of skill can bridge the failure of a single number to fairly handicap boats across light and strong winds — everyone has seen this story and can understand it.

With WRS, Class 40s now find themselves in the top places even in light-air races, because the full power of the polar table has been brought to bear on the problem. The complexity lies in how the TCF is generated, not in how corrected time is calculated: Elapsed Time × TCF is the same principle used by IRC, ORC, ORC Club, SRS and PHRF. WRS gives every boat its own TWS/TWA matrix based on the forecast wind conditions.

For each WRS race, the pecking order under the TCFs is different from APH, and the “muscle memory” of who should win in given conditions no longer applies. With APH it’s easy to anticipate who should win; with WRS it’s much harder — that’s the whole idea. Every handicap on an ORC certificate is related to a forecast wind distribution: the APH forecast was made in 1980, the WRS forecast the night before the race.

We are comfortable with complex systems elsewhere in our lives (maps, streaming, social media) provided they can be used with a few clicks; that is the aim for WRS as well.

3. Why is the final result still a black box?

The WRS process is not a black box: scoring polars are on the certificate, and the predicted tracks and the wind speed and direction used to calculate the Predicted Elapsed Time are published. New analysis tools in RaceFlow, used correctly and communicated well, should eliminate the “black box” perception. The routing itself is run by a third party (PredictWind) using algorithms they maintain for commercial customers; this method is documented on the PredictWind website. But remember, the routing is a means to an end — the deliverable is the wind distribution used by the VPP to calculate the TCF.

4. Can the results be reproduced?

Yes — given the same certificate, course, weather model and settings, the same inputs will produce the same result. Can a competitor check their own score without access to proprietary systems? No — at present there is no published, independent way for a competitor to verify their own WRS TCF outside RaceFlow. The validity of the published TCFs can only be judged by scrutinising the WRS pre-race output.

The situation is analogous to the VPP. The accuracy of the VPP is the best we can do given our resources and the constraint that it should run in two or three minutes per boat. An independent researcher could find more accurate methods to calculate boat speeds, but that does not undermine the fundamental value of the VPP as a tool for handicapping race yachts.

A more useful question might be: can the TCF calculation be assessed before the race? Yes — the WRS output is shared: the forecast used and its timing, the course coordinates, the track data, the time histories of TWS and TWA, current strength and direction, and of course the TCFs. Is this information “correct”? If there are no mistakes, yes. Would another router with different tools get a different answer? Yes — but so what? Our aim is to avoid a boat’s TCF being based on 6–20 knots when the race will actually be sailed in 6–10 knots. The forecast can never be perfect; by focusing too much on the nuances of process, forecast type or routing engine, we risk making the perfect the enemy of the good.

5. What is a reasonable time-to-results?

The speed of delivering results is fundamentally a matter of data-entry efficiency rather than system processing. Under the WRS framework, much like any other handicap method, the TCFs are pre-loaded before the first signal, so the model itself introduces no additional latency. The time between finishing and the final standings is purely a function of how quickly arrival times are recorded and entered into the software.

6. How much complexity is the sport willing to accept?

The issue is transparency, not sophistication. Sailors do not need to understand every line of code, but they should understand where the TCF comes from, why a particular scoring method was chosen, and why it was appropriate for that race. Every ORC TCF is fundamentally based on wind strength and direction because ORC is a VPP-based system; APH, Triple Number, PCS and WRS all come from the same VPP and differ only in the wind distribution used. If a sailor understands the All Purpose Handicap, they should be able to understand WRS just as well, since both are built on the same principle — they only differ in which wind distribution is used to generate the TCF. What we actually need to get better at is explaining, in simple terms, what an ORC TCF really is.

The early proponents of polar-table handicapping would see WRS as the “Holy Grail”: if the weather forecast is perfect, the race is perfectly handicapped.

A common misunderstanding is that a single-number PHRF or IRC rating is equivalent to the ORC APH. Single-number ratings have no mechanism to adjust to different race conditions — they are based on the boat’s physical characteristics, not its predicted performance.

There is an irony to the inherent mistrust of WRS. Every competitor knows that by using APH they have a handicap that is not necessarily appropriate to the wind range they actually encountered during the race. This “horses for courses” situation is acknowledged, and even embraced, as part of the rough and tumble of offshore racing.

Bonus. Where should scoring competence live?

To date, most WRS races have been handled by the ORC team, but the number of races seeking to use WRS has outgrown this capacity — hence the development of the RaceFlow app to widen the pool of competent users.

This has not been without hiccups, but it’s a learning process, and the quality controls are improving. Mistakes will still occur, servers will crash, forecasts will be delayed. It is crucial that the Notice of Race specifies a clear alternative to be used in case of force majeure failures.

If the PRO is satisfied that the WRS process has been completed without error, according to the operating procedure, and that it offers a more appropriate handicap table than the available pre-calculated ones, they can proceed with the full support of ORC. If confidence in the forecast is low, clear fallback procedures should apply — ORC has already introduced these in the Standard Sailing Instructions for major championships, including the upcoming ORC European Championship.

A few things I can’t let go of

First, credit where it’s due. ORC engaged with every question, and their historical framing is strong: APH is, as they put it, a weather forecast made in 1980 that we’ve used for forty years. Nobody defends that with a straight face. The logic behind WRS, handicapping the race that will actually be sailed, is sound.

But four things stand out.

The “No” in question 4. Can a competitor check their own score without access to proprietary systems? ORC answers with one word: no. I appreciate the honesty, but think about what it means. In a sport where sailors do their own penalty turns, the one thing you can’t verify yourself is the number that decides whether you beat the other boat. Published tracks and wind histories let you inspect the output. They don’t let you reproduce it.

“But so what?” Would another routing engine give a different answer? “Yes – but so what?” is the actual quote. I get the point: forecasts are estimates; this is not about perfection. But after a season where a routing discrepancy decided a world championship podium, that’s a lot of weight for a shrug to carry. Sailors aren’t chasing decimals. They want to know if the number that beat them was an artifact of a vendor choice. And that might explain why the most skeptical voices belong to experienced navigators – the people who know exactly how much weather models, grid resolution, and routing settings can move a result.

The question they skipped. I ended by asking what matters most: that the result is correct, understandable, verifiable, or trusted. ORC answered many things, but not that. Maybe because the honest answer is uncomfortable: the four pull in different directions, and WRS optimizes hard for the first one. It’s still the most important question in this debate, and it’s still open.

Whose problem is this solving? Scoring was never the burning issue in offshore racing. We’ve always complained about ratings, but nobody stood on the dock demanding forecast-based correction factors. This has been driven from the top, with real conviction. Fine, most progress starts that way. But when a governing body pushes this hard on a problem the fleet wasn’t loudly complaining about, it’s fair to ask what’s driving the timeline.

One number in passing: 5 races out of 180 needed recalculation this season. ORC offers this as evidence the system works. I’d say no other sport would accept this!?

The debate isn’t only happening on blogs and docks. Several national authorities are sending submissions to ORC’s annual meeting on exactly these questions. The formal machinery of ORC is being asked to reconsider how scoring is chosen, run and communicated. This isn’t a governing body versus a few loud critics. The conversation is happening inside the organization too.

So where does this land?

Better than I expected. ORC admits a systemic gap in quality control, is building validation tools, and is communicating.

The direction is right.

But the finish line hasn’t moved: a scoring system earns trust when a sailor who lost can see why and check it themselves. Until then, every “trust us” rests on the goodwill ORC builds. And conversations like this are how goodwill gets built. So thanks to Thomas and Andy for engaging.

The conversation continues, and judging by this season, it needs to.