When is the new grading list coming out?

[Alex Holowczak]

Looks like too many loses. (ps what about round 6 of the Terafinal?)

I think there were an odd number of players, so there was a half-point bye in there somewhere.

[Mike Gunn]

Roger, I don't think we're at odds over this. My point is that the priority must be to fix the anomalies in the grades which have just been published rather than worry about the "junior problem" (imaginary or not).

[John Upham]

My grapevine of sources inform me that a very recent meeting of the ECF Board decided to attempt no fix of the widespread problem with the new ratings. A phrase similar to "after a year it will sort itself out" was used.

I guess you all know who carries the can for the new ratings...?

Answer : The Director of Home Chess
Cyril Johnson
Tel: 0116 260 9012
[email protected]

I believe that the Manager of Grading & Rating, Dave Thomas reports to the Director of Home Chess.

[James Courtenay]

Personally I was considering filling in one of these forms:

http://www.direct.gov.uk/en/Governmentc ... /DG_067917

[Brian Valentine]

Most of the discussion on this thread has been about the approach to grading Juniors, but the main item for system change was to eliminate stretch. Clearly the recursion approach adds to stretch and not enough data is available to investigate the contribution to the system of this approach.

If the grading committee want to get some constructive suggestions then they will need to make more data available. In order to investigate the recursion influence at a minimum three years data is required – to see what happens to a grade the year after what appears on an anecdotal error. Hence for serious external investigation the committee should make available the “red grades” lists for 2006 and 2007. These need to include those leaving the list, not just the survivors.

What is clear is that previous year’s performance are an important source of information and should not be discarded. If we look at those remaining juniors who were category A or B in both years, their average grade has gone up by nearly 10 points. This is broadly identical to the junior annual addition ( 5 or 10 points depending on age) plus the drift observed for similar seniors. The recursion method appears at most to be reallocating grade points and probably not improving accuracy (however that is defined).

The average grade of the list has fallen 0.3 points year on year. This has come about while, on average, people appearing on both lists have improved. Overall grading has fallen since 1) new entrants (possible re-entrants) are lower grade than those leaving, and 2) Last year there were less new entrants than leavers.

It appears that stretch is emerging again, but more slowly than I was anticipating. It is unclear what is counteracting this natural trend.

On another thread there is a conversation on moving to an Elo system. Elo means different things to different people. There are some aspects that appears better – playing 9 games against RATED players has advantages over a system rating any 9 games, but there are manifest difficulties in converting existing administration.

The methodology, whether ECF or otherwise, sits in a dynamic system with feedback. It cannot be controlled in a way that meets all its objectives. Prime objectives are: 1) placing players in order of strength, 2) avoiding inflation or deflation e.g. for title purposes and 3) individual performance progression. In a different dimension the system can record performance or it can predict future performance.

By implication the ECF has tried to move to a system that better predicts future results. Longer term, this measure cannot be without considering the priorities for those prime objectives. The stretch fix discounts personal progression and the junior fix may foul the order of strength.

This is all difficult stuff. The way forward must involve: deciding what we want from gradings and looking at solutions more holistically.

[Roger de Coverly]

This is all difficult stuff. The way forward must involve: deciding what we want from gradings and looking at solutions more holistically.

More fundamentally the evidence now seems to suggest that the recursion program (or equation solver) is somewhat broken where players score less than 15% or more than 85%. These are approximations as I don't know the true reliability cutoff.

Given
(a) the recursion program can produce seriously low grades - therefore partly or wholly responsible for negative grades
(b) the recursion program was responsible for the claim that the gradings were "deflated" or "spread"
(c) the recursion program is used to calculate the grades of all juniors

Should not the focus be placed on the theoretical and practical downsides of the recursion? EMW has speculated that the lack of a zero sum rule ( play 10 games and the gain for a win is twice that if you play 20) is a contributory factor to instability.
Personally I like perturbation theory - take a high scoring adult who played at least one unrated player. Remove their previous rating and run the recursion routine. Success if it reproduces the previous calculation, failure otherwise. This did after all identify the "Rough" issue.

In fact regardless of the above, the new approach can be rejected on the simple premise that if you only play published players you should be able to calculate your new grade. If the published grade for juniors is ignored, that premise is broken.

[Brian Valentine]

The recursion approach clearly smells rotten.

Before deciding on the fix, I am attempting to develop the ground rules. At the risk claiming to be more fundamental, here are some strands from various threads and analysis of the rating lists:
-The problems highlighted arise mainly from juniors
-Several people are advocating an Elo system, the Elo method at best indirectly addresses this issue
-Elo was designed to rate elite regular players not the ability range with which the ECF covers
- 45% of Cat E players on the 2008 list have no grade in 2009, these are most likely to have recursive grades (is it worthwhile having the E category?)
- The junior annual additions are the main source of inflation in the system, which because of leaver and joiner effects is naturally deflationary
-Both Elo and basic ecf seem to assume that recent performance is a guide to current ability. The ecf recursion addition seems to accept that juniors are different and that in some way the system a poor guide for first time grading. Possibly true, but I have seen no theoretical underpin or which objective of thesyatem that the graders are aiming at.

The stretch fix arose to address one perceived problem and has had serious side effects. The recursion approach needs fixing. Many aspects of the Elo system look attractive, others have weaknesses. The fix should be directionally consistent with what the membership wants from the system. We do needto look at the system holistically.

[Matthew Turner]

I haven't been able to follow the whole of this discussion (both because of it's length and some of the technical detail), however I would make the follwing observations.

1. We cannot tell how bad (or good) the grading list is at the moment. It does not really matter if Callum Kilpatrick is graded at 190, 210, 230 or 250. What matters is players like Neill Cooper's son, whose grade changes what events he is eligible for. It will take a while to see how widespread these 'anomalies' are.

2. The grading team seem to have one far beyond their remit. Howard Grist seemed to be suggesting that the grading computer was producing a junior grade that would be accurate in January. Surely their computer isn't powerful enough to predict the course of human development otherwise everyone's grade would be 42?

3. Let's remember what a grading system is for it ranks players. There was no need to fix the 'problem' that players graded ten points higher than there opponent only scored 58%. This still meant the ranking was correct and it was a self-correcting error.

4. Finally, let's hope that juniors don't give up because they have to play in Opens rather than Minors, or because their grade has mysteriously dropped 50 points. As organisers, lets try to be sensitive.

[E Michael White]

Roger

Should not the focus be placed on the theoretical and practical downsides of the recursion? EMW has speculated that the lack of a zero sum rule ( play 10 games and the gain for a win is twice that if you play 20) is a contributory factor to instability.
Personally I like perturbation theory - take a high scoring adult who played at least one unrated player. Remove their previous rating and run the recursion routine. Success if it reproduces the previous calculation, failure otherwise. This did after all identify the "Rough" issue.

I have been looking at this again recently and believe I can account for most of the issues; if I have time will post something later.

The mathematical iterative process is not really the place to look for errors as it just solves linear simultaneous equations in a different way. The main issues are with grading players effectively on an insufficient number of games and attempting to backsolve to find grades of ungraded players at the beginning of the rating period. This latter method is also written into FIDE rules for certain events !

[Roger de Coverly]

address one perceived problem

I've never been convinced there was much of a problem in the first place - at least not in the top quartile. I think the quoted evidence of under performance was collected by running the recursion routine and comparing it to the live grades. If we now accept that the recursion has flaws at the extremities, this casts some doubt on the value of this investigation. Attempts to find alternative evidence of stretch for example changes in the average grade or widening of differences between the grandmaster level and the 170s level only seemed to indicate relative stability. Looking at player histories also seemed to indicate stability for the more active players in the top quartile. Is there evidence of stretch or deflation that has been collected without using the recursion routine in any shape or form? I would dismiss comparisons with international ratings as evidence because that also indicates with equal or greater validity inflation in the international system.

It was apparent that there were problems appearing in the lower quartiles for example older players at 120 and under seemed to be drifting downwards and there were issues arising with negative and very low grades. These seemed to start after the recursion routine went live. In the same way that the recursion can propel a third division London League player into the top 50, it can also give a misleadingly low strength estimates for players doing badly in their first tournament or two.

I think the graders must now accept that the recursion should be used with caution. This may indicate a partial return to the concept of "grader's estimate" for that initial grade and partial use of converted grades from other systems where available.

What's to be done for this year, I'm not sure. What are the alternatives to restoring the previous system? The problems with extreme results will not go away, nor will the lack of usefulness of having a subset of published grades that you cannot use for performance calculations.

On the original plan, a "hard" set of "red grades" was supposed to be available in August 2008 so we had a whole year of parallel running. In the event we didn't get the 2008 parallels until August 2009 and this was coupled with major and controversial changes to the methodology. Within a couple of days, potential flaws with the approach have been identified. We should have been there in August 2008.

[Roger de Coverly]

The mathematical iterative process is not really the place to look for errors as it just solves linear simultaneous equations in a different way. The main issues are with grading players effectively on an insufficient number of games and attempting to backsolve to find grades of ungraded players at the beginning of the rating period

I think we agree that the back-solving process needs to be used with caution particularly with extreme results. In the limit all we know about a player that scores 100% against 150s is that their grade should be at least 190.

This latter method is also written into FIDE rules for certain events

. It's a very new rule as previously games against unrated players didn't count at all in the international system. I haven't noticed it being applied in practice yet, but I think it's a single pass system. The idea being to include rating changes for the games between rated and unrated. They just take the provisional rating and apply it to the rated players. If they try to do unrated v unrated, they will hit the same convergence problems.

FIDE already have an all play all rule which lets you get a rating by playing unrated players. Sean's Uxbridge events are taking advantage of this on this very day. You need 4 rated players in 10 and then everyone who scores at least something (1 point I think) gets a rating.

[Roger de Coverly]

graded at 190, 210, 230 or 250.

It may of course affect the pairings and board orders for non-FIDE events. The days have passed when a high British grade was a title qualifier or criteria for international selection. However it does remain as a qualification route to the 2010 British that an ECF grade of 218 suffices. So the big hitter London League player will qualify because he was both ungraded and played an ungraded player whereas someone with exactly the same performance who was either already graded or only played graded opposition would not. I don't think we want that sort of randomness in the grading system.

[Richard Bates]

It's a very new rule as previously games against unrated players didn't count at all in the international system. I haven't noticed it being applied in practice yet, but I think it's a single pass system. The idea being to include rating changes for the games between rated and unrated. They just take the provisional rating and apply it to the rated players. If they try to do unrated v unrated, they will hit the same convergence problems.

FIDE already have an all play all rule which lets you get a rating by playing unrated players. Sean's Uxbridge events are taking advantage of this on this very day. You need 4 rated players in 10 and then everyone who scores at least something (1 point I think) gets a rating.

I seriously worry about what FIDE are doing with their rating system and the new "everyone can have a rating down to 1200" changes. Leaving aside the effect this will have on juniors getting ratings "too early", I don't think it will be long before big International opens become further undermined. The problems of accelerated pairing systems have been discussed elsewhere, I fear in future that Organisers will have to put "minimum rating" requirements on their Opens as these problems begin to multiply with the bulk of entries increasingly in the bottom quartile.

"Get a rating, have a chance of playing a GM, and win a grading prize" will encourage significant entries which just wouldn't have happened before. Good for Congress finances perhaps, but terrible for the tournaments themselves. Even worse if the rating system cannot be relied upon, with large number of ratings essentially based on one isolated rating performance (not even produced by games solely against rated players).

[Ben Purton]

It's a very new rule as previously games against unrated players didn't count at all in the international system. I haven't noticed it being applied in practice yet, but I think it's a single pass system. The idea being to include rating changes for the games between rated and unrated. They just take the provisional rating and apply it to the rated players. If they try to do unrated v unrated, they will hit the same convergence problems.

FIDE already have an all play all rule which lets you get a rating by playing unrated players. Sean's Uxbridge events are taking advantage of this on this very day. You need 4 rated players in 10 and then everyone who scores at least something (1 point I think) gets a rating.

I seriously worry about what FIDE are doing with their rating system and the new "everyone can have a rating down to 1200" changes. Leaving aside the effect this will have on juniors getting ratings "too early", I don't think it will be long before big International opens become further undermined. The problems of accelerated pairing systems have been discussed elsewhere, I fear in future that Organisers will have to put "minimum rating" requirements on their Opens as these problems begin to multiply with the bulk of entries increasingly in the bottom quartile.

"Get a rating, have a chance of playing a GM, and win a grading prize" will encourage significant entries which just wouldn't have happened before. Good for Congress finances perhaps, but terrible for the tournaments themselves. Even worse if the rating system cannot be relied upon, with large number of ratings essentially based on one isolated rating performance (not even produced by games solely against rated players).

Czech Open already do this

[Richard Bates]

Czech Open already do this

Minimum rating, you mean?