How can we fix the ORC WRS?

by Nils Melsom Kristensen (nilsmk@duck.com)

Weather routing scoring could have been one of the best things that has happened to handicap racing in ages, but unfortunately it seems to be failing. Surprise surprise, it’s not primarily due to unreliable weather models, it’s a fundamental flaw in the weather routing module. Here is my humble attempt at explaining the problem in a way that is understandable to most sailors. All the work below is based on the recent Færder race in Norway (https://manage2sail.com/no/event/31734e24-397d-453e-85ce-c4cc544000e7#!/).

Disclaimer: I was sailing the race onboard the X-41 Saillogic. I also work as a scientist at MET Norway and produced the official weather briefing for the event, but I am not directly connected to any of the weather models used in the WRS. So even if I might have a personal interest in the case, my only agenda is to fix the WRS, which is currently not working as intended.

To explain this, I will use the case of four X-41 sailing in the same class, same course, same start. Here are some facts about the boats and rating. All percentage differences below use Saillogic as reference.

It is expected to have some differences in the rating relatively between boats when moving from APH to WRS, since the polar diagrams are not exactly the same. However, to have One Design boats (like the X-41) with only slightly different sail configurations to change by 8% when moving from one system to another is something that should be grounds for a more thorough examination.

Let's first look at the measurement certificates (link in table above). The VPP performance number (polars) can be found below.

Then, let us compare these two. See both absolute and relative differences in speed in the tables below. Positive numbers means Saillogic is faster, negative numbers means BarX is faster.

As we can see, Saillogic is generally faster than BarX, which is also expected since the APH of BarX is 0.8% less than Saillogic. The largest relative difference in predicted speed is 8.8%. This is reaching in 24kt TWS. The Færder race saw a maximum wind of about 16kt for a few hours, while the boats were sailing upwind or very tight reaching at TWA 50-90 deg.

According to the WRS routing prediction for Saillogic (https://raceflow.orc.org/yacht.php?wrskey=dfb07caf0f5848f783dfc2703df576af&refno=03440004S0S), the maximum wind used is 10.4kt. For BarX (https://raceflow.orc.org/yacht.php?wrskey=dfb07caf0f5848f783dfc2703df576af&refno=03440004RVQ), the maximum wind is 10.1kt.

Then, looking at the relative differences in less than 10kt TWS above, we see the maximum difference is 5.1% in 4kt TWS.

Again looking at the WRS prediction, we see both boats should spend no more than about 2 hours in wind below 5kt, and when taking into account that the predicted total sailing time for Saillogic is about 16 hours, and BarX about 17.5 hours, this accounts for very small parts of the race, and no difference in performance between the boats for the wind range used in WRS is remotely close to the 8.8% difference in rating.

So, let us continue the search.

As was said during the official weather briefing, the weather was very uncertain due to the passing of a small local low pressure system that was passing over the race course during the race. In hindsight, the weather development was very close to the latest forecast that was used in the WRS scoring for the race. The low probably passed a bit further south than expected, giving a more stable breeze than expected, but directions were very close to the forecasted weather. It is common knowledge that forecasting weather is not a precise science due to a number of reasons, something every meteorologist is open and honest about. However, the forecast was generally in good agreement with the actual weather for the race. And, the real weather has no impact on the rating we have discussed so far…

There is no perfect forecast, no perfect weather routing, and no perfect rating system, but the WRS should theoretically be one of the better “simple” systems we can come up with, and it is therefore very surprising that the rating behaves so unpredictable.

Let us dig deeper…

If we look at the WRS output for the Færder race (https://raceflow.orc.org/routing.php?wrskey=dfb07caf0f5848f783dfc2703df576af), and the four X-41’s, we can view their predicted courses in the map below.

Unfortunately, there is a lot of clutter from the mark names, but trust me when I say that the predicted optimal routes align very well.

So, why would the WRS predict that almost identical boats, sailing the same route, in the same wind perform so differently?

If we zoom in on the predicted routes, we see something strange. This is the crux of the matter. We can clearly see that the wind barbs are placed at different positions along the optimal routes. When using a grid-based routine algorithm, these wind barbs are usually placed at the start and end of the route segments, so we assume the same is the case here. In the example below, I have zoomed in on the part of the course between Moss and almost Færder. Now, take a close look at the “brownish” optimal route, and then the white, light grey and purple. The routes on the left are the ones going south, while the ones on the right are the ones returning north. On the southward routes, there are clearly very few wind barbs on the brown route. In fact, going south in this area, I can only spot two (I tried to mark them with red dots), whereas the purple have at least eight.

Why does this matter, you say? Well it does. The wind at the start and end of the routing segment is the wind used to calculate the time spent for that segment based on the boat’s performance prediction. Either the wind at the start of the segment is used on the entire segment, or the wind is linearly interpolated between the two.

If the wind was stable, and did not change in time or space, then it would not matter how many of these routing points you had, since the sum of the predicted times for each segment would be the same as the value for the very long segment. But when the wind is changing, we get a kind of aliasing that would give a different sum of time for the same route depending on where, and how many routing segments you have. See a simplified example in the “Data and analysis”-section below.

Brown: BarX, Purple: Xanadu, White: Saillogic, Light grey: Gottix

This type of behaviour in the routing algorithm is common knowledge among offshore navigators who use grid based routing software. When these long jumps occur, it is common to make small changes in the routing settings to try to avoid them. Why they happen, I don’t really know, but I guess there is a bug somewhere… As far as I know, these jumps do not appear commonly in isochronal based routing algorithms.

So, where do we go from here, how do we fix it? Well, only the provider of the routing can do that, but they should start by ensuring that the routing segments have a more evenly distributed length, and that they are not too long. Additionally, it would be wise of the ORC to admit the problem, and how this may have influenced the results of races that have been scored using WRS.

As I final note, even if I am very certain that this is the problem, I can be wrong, and I am definitely open to other explanations if anyone has one.

Data and analysis

Comparing the yachts that sailed the longest course, APH on x-axis, WRS on y-axis. Red line is linear regression, while the yellow are +/- 1 standard deviation and the green +/- 2 standard deviations. Yachts that fall below the red line have “better WRS than APH” , while the yachts above the red line “have worse”. Boats in blue generally behave as expected, while the red and orange are some that should be looked into.

Overall, it is expected that relatively similar boats that are considered “all-round boats”, should have a similar relative relation between them if it is APH or WRS.

Average leg length (of the predicted route) vs absolute deviation from the regression line, minimum leg length vs abs_deviation and maximum leg length vs abs_deviation. See e.g. BarX has the longest leg of all yachts, over 18nm on a 88nm race…

Same as above, but with actual deviation, not absolute (negative means “better rating”, positive means “worse rating”).

Statistics of Leg Distance length vs TWS to look for patterns. There seems to be no significant correlation, other than a tendency for the very long legs to only occur in low wind.

Histograms summarizing all the routings wrt. leg distance length and TWS.

Above: TWS used by the routing algorithm (from RaceFlow) for the 4 almost identical X-41s. This is based on the same forecast, so forecast error is still not relevant. The only difference is the difference in routing discretization.

Above: We calculate the TWS difference between the start and the end of each routing segment for each boat. Then we multiply the TWS_delta with the segment distance, and summarize for each boat. The figure is interesting and largely indicates who 'loses' and 'wins' due to the error I am pointing out. Boats below the horizontal line have a lower rating than expected based on APH, while boats above have a higher rating. Boats to the right of the vertical line have to a large extent sailed these 'long routing segments' in increasing wind, meaning they have been calculated with a lower wind speed than what it probably should have been. Boats to the left have sailed multiple or long legs in decreasing wind. This is not a perfect way to quantify the error, but I believe we should be able to say that the boats to the right of the vertical line have benefited from the error, while those to the left have been at a disadvantage. The further from the vertical line, the greater the disadvantage/advantage. It seems to be systematic that long legs with increasing wind give favourable ratings.

Summary and conclusion…

To summarize, and try to draw some simple conclusion, I will say that due to different discretization of the routes for different (or almost identical) yachts in a race scored using the WRS, the routing algorithm influences the rating. It’s like calculating identical yachts on slightly different wind models. If this creates a problem for almost identical boats, it is fair to assume that exactly the same occurs for boats that are “different from each other".

Weather model forecast error is another discussion, but it is not relevant to even start to discuss that, and how it influences the rating, since the errors we are discussing now have nothing to do with forecast error. It is simply a numerical effect.

If the ORC does not acknowledge this, and ask the routing provider to fix it, they are effectively throwing the baby out with the bathwater. It’s like saying that errors in the VPP are not important, because, in the end, there is no perfect truth.

There is no such thing as a perfect rating system, but ignoring large errors in the system when they are pointed out, is, at best, just ignorant…

I have been sailing handicap races scored on IMS and ORC for around 30 years, and have gained insight and trust in the system over the years. In my opinion, ORC is the best rating system that exists. But I think what we see now could very well be the start of the end for at least the WRS, if not the entire ORC rule, if we don’t see an effort from the ORC to accept the problem, and come up with a public admission and plan for a fix…

But hey, I’m just a simple guy, sailing because I think it’s fun, so what do I know :P

More data and code

Simple example of effect of different routing resolution on WRS rating:

https://docs.google.com/spreadsheets/d/e/2PACX-1vR0XTamnfB4ASnCiyrtejX4AeBaUn67zE5YTxlfuQAr__FEM2NlBscRINnyCgK4bjVPzdQvp_b-qJHM/pubhtml?gid=0&single=true 

Quick visual comparison of the weather model used for WRS and the following analyses:

https://docs.google.com/document/d/e/2PACX-1vSmH_D5L07LWlfqlDoiW-uPncPU8zhUoS2OyvIFemVPMTSyEaNEArw3vX1pz2hMDds0MH-F-KZkx6gX/pub 

More details of how the WRS compares to APH for the Færder race:

https://docs.google.com/spreadsheets/d/e/2PACX-1vTo8K6r3ojkzwydFfNcimHww2UlMyF3ikxBiN7AcQtOUgHYM8Ij3DrNrV25bhuLkPOnU3GL4Zg9ecH5/pubhtml?gid=2053626941&single=true 

Code used in most of this analysis:

compare_aph_wrs.ipynb