GRADING ANOMALIES

[Robert Jurjevic]

Please note that a new version of the document is out now (version 09/06/2009 1.11)...

http://www.ecforum.org.uk/viewtopic.php ... 149#p10149

http://www.jurjevic.org.uk/chess/grade/ ... malies.htm

This is pretty much a definitive paper proving (mathematically and on the examples) why present grading system GS is stretching the grades and proposing three alternative grading systems, ÉGS6, ÉGS5 and AGS3, which would (should) not stretch the grades.

Now, as new corrected grades are about to come into effect, it could be the right time to adopt one of the proposed grading systems in order to prevent grade stretching in the future.

AGS3 is the simplest one and in my opinion should be acceptable even to the most conservative minds, as it only requires changing the grading rule to:

Rule 2b: For a win you score average grade plus 25; for a draw, average grade; and for a loss, average grade minus 25. Average grade is half of the sum of your and your opponent's grade. Note that, if your opponent's grade differs from yours by more than 40 points, it is taken to be exactly 40 points above (or below) yours. At the end of the season an average of points-per-game is taken, and that is your new grade.

I would personally vote for ÉGS6, as it has FIDE's Élo relationship for 'p = f(d)' and addresses the problem of less established (or in extreme case estimated) grades (the system changes the grades of less active players more than those of more active players, so that in extreme case ungraded players do not affect the grades of graded players).

Quick section guide: Factors 'k' and grade stretching (shows that factors 'k' are the cause of the stretching), "Equal grade for equal performance" (shows why "equal grade for equal performance" rule should be abandoned in favour of the system which does not stretch the grades), Factors 'k' and total system grade (shows that factors 'k' determine if the system preserves total grade), In a nutshell (describes in short what is the cause of grade stretching), Mathematical proof (mathematically proves the condition which ought to be satisfied in order for a system not to stretch nor shirk the grades), AGS3 (introduces the simplest system which does not stretch nor shrink the grades), ÉGS5 and ÉGS6 (introduces two new systems which improve on ÉGS and ÉGS2, ÉGS5 and ÉGS6 use 'p = f(d)' matching the FIDE's choice exactly), Which 'p = f(d)' (describes why FIDE's choice of 'p = f(d)' is regarded as the best), Variable factor 'k' (discusses variable factor 'k' used in ÉGS6), ECF grade vs FIDE rating scale (describes why ECF grade scale makes more sense to me), Replacing GS (discusses the candidates for replacing the current grading system), "Junior problem" (mentions the "junior problem"), Estimated effect on stretching (estimates the effect of key factors on grade stretching), The formulae (lists all relevant grading formulae).

Thanks a lot.

[Brian Valentine]

Robert,
The original entry by Chris Majer was based on a paper which summarised the issue to address as follows:
“1) The actual performance of players did not appear to correspond to the theoretical performance. This means that whereas for a grading difference of 25 points the stronger player should score 75%, his actual score is more like 68%.
2) The problem appears to have been present at least since the introduction of the submission of individual results in 2000 and probably since 1995, which is the earliest date for which an electronic record of grades is available to the team. It has been suggested that it stems from the extension of the grading system to players graded below 175 (in present terms) in the 1960s. “ (extracted from the ecf website)

Sean Hewitt conducted some analysis which demonstrated a stretch in the system. This analysis lead to the “new grades” to be introduced next year. In the original entry in this topic Chris also set out a 6 part plan. To my knowledge only the final part has been fully completed.

Going back to first principles we need to agree what a grading system is for. Here is a good summary.

“Chess rating systems have many practical uses. For pairing purposes in tournaments, a tournament director should have some idea which players are considered the most likely candidates to win the tournament so the director can effectively avoid pairing them against each other during the earlier rounds of the tournament. Ratings are also used for tournament sectioning and prize eligibility; a section in a tournament may only allow players of a specified rating range to compete for section prizes. Ratings can also be used as a qualifying system for elite tournaments or events; invitation to compete in the US closed championships and to compete in the us Olympiad team are based in part on players’ USCF ratings. The current “title” systems used by some chess federations base their title qualifications on the overall strength of tournament participants as measured by their ratings. But probably the most useful service of the rating system is that it allows competitors at all levels to monitor their progress as they become better chess players” (from: Rating the Chess Rating System by Glickman and Jones)

Hence the first requirement of a system is the ranking of competitors in order of strength. The second is to avoid the inflation or deflation both for the consistency over time of title awards and monitoring of individual progress in chess.

The stretch issue is quite important subset of the investigations of inflation and deflation but does not affect ranking. I’m struggling to see why you are spending so much effort on radical changes to address this sideline. Especially when you appear to be avoiding the point that stretch (at some level) must remain in all the alternatives you propose.

There could be a discussion on the “30 games” rule – your k factor discussion, but not starting from where you are.

There have also been threads in the discussion about “fairness” or “penalising” these are not appropriate concepts. We are trying to get the rankings right across all competitors, in a way which works now and makes cross temporal comparison possible.

Sudden changes in a grading system have to be really worthwhile to overcome what Glickman suggests is its most useful service.

[Roger de Coverly]

This means that whereas for a grading difference of 25 points the stronger player should score 75%, his actual score is more like 68%.

I don't know whether I've ever seen a satisfactory explanation of how this was measured. But assuming it to be true, why doesn't this act to counter stretch? You could define stretch as an effect whereby the "top" and "bottom" of a rating list are further apart from the "average player" in the middle of the distribution than they should be. For example the revised list seems to move the 230 graded GM about 30 points closer to the centre of the distribution. My point is that if the 230 GM is only scoring 68% against the 205 FM, why doesn't this counter stretch by reducing his grade back towards the middle? Grading systems by their nature and longevity are supposedly self correcting.

Sudden changes in a grading system have to be really worthwhile to overcome what Glickman suggests is its most useful service.

It remains my opinion that reworking all the grades is the very last option the grading team should have considered instead of, as it seems, the first. The empirical evidence of the wallchart should have been given much more weight. The grades of players in open swisses are much the same as they've always been from the GMs at 230 plus down to the rank and file in the 160s. The playing standard (in openings and early middle games at least) is probably higher. But the standard needed to be a GM is higher as well.

[Robert Jurjevic]

Brian, thank you for your reply.

The original entry by Chris Majer was based on a paper which summarised the issue to address as follows:
“1) The actual performance of players did not appear to correspond to the theoretical performance. This means that whereas for a grading difference of 25 points the stronger player should score 75%, his actual score is more like 68%.
2) The problem appears to have been present at least since the introduction of the submission of individual results in 2000 and probably since 1995, which is the earliest date for which an electronic record of grades is available to the team. It has been suggested that it stems from the extension of the grading system to players graded below 175 (in present terms) in the 1960s. “ (extracted from the ecf website)

My belief is that I have found the main reason for grade stretching, a simple summary of the finding can be found in In a nutshell section at...

http://www.jurjevic.org.uk/chess/grade/ ... malies.htm

The stretch issue is quite important subset of the investigations of inflation and deflation but does not affect ranking. I’m struggling to see why you are spending so much effort on radical changes to address this sideline. Especially when you appear to be avoiding the point that stretch (at some level) must remain in all the alternatives you propose.

In my opinion grade stretching is not a sideline, say my grade will go from 90 to 120ish, it looks to me as a substantial correction to my grade. Maybe there are other factors which may cause grade inflation and/ or deflation (and other anomalies), but in my opinion, one should address the most obvious one (and the one which contributes to grade stretching the most) first.

In my opinion, a minimum change required to practically stop grade stretching (though possibly not other grade anomalies for which I believe would be small relative to the present grade stretching) would be to change the current grading rule to:

Rule 2b: For a win you score average grade plus 25; for a draw, average grade; and for a loss, average grade minus 25. Average grade is half of the sum of your and your opponent's grade. Note that, if your opponent's grade differs from yours by more than 40 points, it is taken to be exactly 40 points above (or below) yours. At the end of the season an average of points-per-game is taken, and that is your new grade.

(i.e. to adopt AGS3), though other proposed grading systems have advantages (over AGS3), as summarized in the table 1 below (ÉGS6 would be the best in my opinion):

Code: Select all

--------------------------------------------------------------
grading   stretches  uses FIDE   changes less     preserves 
system    grades     'p = f(d)'  trusted grades   total system
          ('k')      (yellow)    more rapidly     grade
--------------------------------------------------------------
GS        yes        no          no               yes
AGS3      no         no          no               yes 
ÉGS5      no         yes         no               yes
ÉGS6      no         yes         yes              no
--------------------------------------------------------------

Table 1: Main differences between GS (current Grading System), AGS3 (Amended Grading System three), ÉGS5 (Élo Grading System five) and ÉGS6 (Élo Grading System six).

If you wish you could look at the example given in Factors 'k' and grade stretching section at...

http://www.ecforum.org.uk/viewtopic.php ... 149#p10149

where it has been shown that the grades were stretched for 30 grading points by the current grading system. Grade stretching is proportional to the difference between expected and actual performance '|p - q|' and it happens all the time when '|p - q| > 0' and it accumulates as the seasons pass.

Thanks a lot.

[Roger de Coverly]

In my opinion grade stretching is not a sideline, say my grade will go from 90 to 120ish, it looks to me as a substantial correction to my grade

That's only because instead of taking 30 points off the GMs and leaving the "average" player unchanged, they choose to compress the system by leaving the GMs more or less unaltered and increasing everyone else.

I will become a "new" 180, thus I should score about 80% against new 150s who should score about 80% against you. What's going to happen if the old relationships were after all correct? They were stable in that the relationship (grading distance) between my part of the world in the 170s and the GMs in the 230s has been much the same for most of the available on-line history and was much the same back in the 70s and 80s as well. My hypothesis is that the enhancements to the grade of the "average" player are likely to be permanent because of their weight of numbers but that the compression of the system will unwind and result in inflation of the top grades over the next few seasons.

[Robert Jurjevic]

In my opinion grade stretching is not a sideline, say my grade will go from 90 to 120ish, it looks to me as a substantial correction to my grade

That's only because instead of taking 30 points off the GMs and leaving the "average" player unchanged, they choose to compress the system by leaving the GMs more or less unaltered and increasing everyone else.

Should I expect roughly the same correction of my 'new' grade in say five years time?

[Roger de Coverly]

Should I expect roughly the same correction of my 'new' grade in say five years time?

I doubt it - because the ECF system will have been so discredited by the rampant inflation resulting from this year's revaluation that there will be a move to scrap it and replace it with a national Elo type system

[David Shepherd]

Based on the current system with the points awarded the player with the higher grade who draws is treated as being worse than the person with the lower grade he draws with

Being pedantic (and why not), the ECF system "works" on the basis that you add up the previously published grades of your opponents, then you add 50* the excess of wins over losses. Your new grade is that total divided by the number of games. No statements are made or needed regarding the results of individual games. So (assuming at least 30 games) your average opposition is 150 and you score 50%, your new grade is 150. If your average opposition is 150 and you score 75%, your new grade is 175. If you only make 25%, it's 125. It doesn't matter if your 50% score includes a win against a player rated 175 and a loss against a player rated 125. Provided the 40 point rule doesn't apply and you play at least 30 games, your own previously published grade isn't a parameter. I think the principle of equal grade for equal performance is a correct one - in the context of establishing a performance over a year long season.

Thanks Roger - I understand your point, and see that the current system should work ok if sufficient games are played, unfortunately this is often not the case. There is also an inbuilt assumption that the 150 players are 150 players, but at the lower end of the grading range this will not be the case hence the junior increment, but maybe this should have been extended to adults in certain circumstances for example adults who have not been graded before could get 5 or 10 points added for the first couple of years they are graded.

[Robert Jurjevic]

Based on the current system with the points awarded the player with the higher grade who draws is treated as being worse than the person with the lower grade he draws with

Being pedantic (and why not), the ECF system "works" on the basis that you add up the previously published grades of your opponents, then you add 50* the excess of wins over losses. Your new grade is that total divided by the number of games. No statements are made or needed regarding the results of individual games. So (assuming at least 30 games) your average opposition is 150 and you score 50%, your new grade is 150. If your average opposition is 150 and you score 75%, your new grade is 175. If you only make 25%, it's 125. It doesn't matter if your 50% score includes a win against a player rated 175 and a loss against a player rated 125. Provided the 40 point rule doesn't apply and you play at least 30 games, your own previously published grade isn't a parameter. I think the principle of equal grade for equal performance is a correct one - in the context of establishing a performance over a year long season.

Unfortunately, I have found that apparently there are two options: one could opt either for a system which obeys "equal grade for equal performance" rule and stretches the grades or a system which does not obey "equal grade for equal performance" rule and neither stretches nor shrinks the grades.

Please find below copied "Equal grade for equal performance" section from...

http://www.ecforum.org.uk/viewtopic.php ... 149#p10149

"Equal grade for equal performance"...

Systems which neither stretch nor shrink the grades do not obey the rule which is known as "equal grade for equal performance" (say if you have a 130 player who scores 50% against a pool of 160 players "equal grade for equal performance" rule requires that the 130 player becomes a 160 player; according to the systems which neither stretch nor shrink the grades the 130 player becomes approximately a 145 player).

So it would seem that one could opt either for a system which obeys "equal grade for equal performance" rule and stretches the grades or a system which does not obey "equal grade for equal performance" rule and neither stretches nor shrinks the grades.

Let us assume that in the above "equal grade for equal performance" example all players (the 130 player and all pool players) were ungraded (their grades were estimated) and that the 130 player played 300 games during a course of a season and that each player in the pool (there are 10 players in the pool) played 30 games against the 130 player. Then, taking into account only the 300 games the pool players played against the 130 player, it follows (from the relationships 'p = f(d)') that (at the end of the season) the 130 player should be regarded approximately equally strong as the pool of players (he played).

"Equal grade for equal performance" rule requires that new grade of the 130 player is 160 (assuming that the pool grade stays approximately 160).

Taking into account only the 300 games the pool players played against the 130 player, a system which neither stretches nor shrinks the grades requires that new grade of both the 130 player and the pool is approximately 145.

The requirement of the system which neither stretches nor shrinks the grades makes a lot of sense (to me). As the 130 player should be rewarded, so the pool players should be penalized, it would be unfair that the pool grade is not affected by the 300 games the players played against the 130 player. If the pool players (during the course of the season) played only 300 games against the 130 player the pool grade should have dropped to approximately 145. The fact that the pool players might have played other games during the course of the season (in my opinion) should not affect a decision on how much they should be penalized for their 300 games played against the 130 player, which in turn determines how much the 130 player should be rewarded (i.e. the 130 player for his 300 games against the pool players should not have been given higher grade than approximately 145, regardless how good the pool players might have performed against the other players).

The above argument is enough for me to claim that "equal grade for equal performance" rule is unsound and should be abandoned in favour of a system which neither stretches nor shrinks the grades.

Note: All of the arguments above would hold if the grades were established (I thought that a relatively large difference between expected and actual performance in the examples above would be easier accepted if it was assumed that the grades were estimated).

Please note that if the pool of players (during the course of the season) played only 300 games against the 130 player, according to the current grading system, the 130 player's grade would rise to 160 and the pool's grade would drop to 130, so the grades would be stretched for 30 grading points (please note that the 130 player and the pool should be regarded equally strong), as according to the system which does not stretch (nor shrink the grades) the grades of both, the 130 player and the pool, should become approximately 145.

[Roger de Coverly]

There is also an inbuilt assumption that the 150 players are 150 players, but at the lower end of the grading range this will not be the case hence the junior increment,

I wouldn't disagree - I think any grading system struggles to cope with "below average" players simply because it should be relatively easy to improve. The junior increment also applies to "strong" players as well. I think it was originally introduced around the late sixties at 5 points. The shear number of strong teenagers in those days was creating a perceived deflationary effect. Later in the seventies, it was increased to 10 points. By the late eighties, it was blamed for inflation of the mean grade and scaled back to the age related scale. On the recalculated grades, the proposal is to scrap it for over 14s, but increase it for younger players.

but maybe this should have been extended to adults in certain circumstances for example adults who have not been graded before could get 5 or 10 points added for the first couple of years they are graded.

It's a perfectly valid idea provided you could only apply it to genuine new players and not to players who were new to English chess but not new to competitive chess. One issue is that a player who has a performance of 50 in their first season and 100 in their next probably comes out with a lower grade than a new player with the same second year performance. This is because of the 40 point rule "remembering" their performance of 50. You could also discard previous season's performance at 20 games (instead of 30) for new players.


Many of these empirical issues could and in my opinion should have been investigated before they embarked on destabilising the grading system and grading limited competitions.


There's an impliedf assumption in the grading system that there's a background of players whose strength and results don't change much from one year to the next. These provide a benchmark against whom you can measure improving players. So If you score 75% against a field of 150 players, your new rating will be 175 (for sufficient games) and theirs will be 149. However if there is also a passing 175 who only scores 50% against the 150 field, they (the 150s) get their points back.

[Alex Holowczak]

This is going to sound like pouring cold water over it all, but does it really matter?

The grades were stretched, which isn't desirable, but having gone to a lot of effort, it seems all that can be done is reduce the stretching.

When I enter a tournament, I don't really care if my opponent is 104 or 106. There is going to be virtually no difference between the two in terms of my chances of beating them. Indeed, at that level, he might be a 140 at a d4 opening, but a 80 at an e4 opening (or equivalent). So, depending on the opponent, a 120 may have greater expertise at one opening than someone else graded 120, and get better results accordingly, even if he is equally good.

Furthermore, team games are used in the grading system. In a 6-board match, if it's 3-2, the person on 3 would be tempted to abandon all plans of trying to win the game, and just make sure of a draw to give his team the win. This will artificially reduce his grade.

As long as each player is roughly their grade, I don't see the problem. The grades were stretched, which was mainly due to juniors (I believe) and counting games from three years ago. A junior can increase rapidly over 3 years.

In my opinion, a less rigid way of doing it would be to use a system similar to that used in Go, but Chess already has it to an extent. Why not have the top graded players graded 1, then the next down graded 2 etc. down to about 20. Everyone fits on to this list, beginners (no graded games) start at grade b (for beginner). Then, you have to achieve a series of norms to progress up the system to get to the next grade. English GMs would be graded 1, IMs 2 and FMs 3, then the rest would hopefully fall into place behind them. At tournaments, the sections wouldn't be U130 or something, they'd be U10. You could jump up as many tiers at once as you like, too, so long as you meet the norms. Since you couldn't lose your grade, having low graded players wouldn't annoy people in higher sections (I read elsewhere on here that that was an issue with some people).

[Brian Valentine]

Robert,
I had read your summary before responding last time. I think the agenda is wrong. However if you want people to follow it then can I make some suggestions on the early part.

1. There is no evidence that stretching is taking place. There is evidence that high graded players tend to under perform and lower graded players outperform. This is a weakness in all grading systems and may even have been constant over a long period, if not since inception. Hence I think you must define precisely what YOU mean by stretching. I think your mathematical proofs hide the real problems in player universes, rather than a simple match situation, but I can't be sure.

2. It would help if you could append summaries of all your alternative grading system.

3. In one paragraph you state: "Let us assume that two ungraded players both with estimated grade of 100 play a match of 30 games during a course of a season and that one of the players scores 80%. Then, it follows (from the relationships 'p = f(d)') that (at the end of the season) one of the players should be regarded stronger (than the other) for approximately 30 grading points.". This seems to be the foundation of your stretching argument. I think you are mis-representing the graph which is performance against a 100 player (in your world). As you state the ecf system would rate the diffence as 60 points. However if there were only 2 players then you wouldn't need a rating system as score -to- date would be a sound ranking system.

Looking at the recent thread I think we need a definition for "equal grade for equal performance" since I suspect yours is different from Roger's idea.

[Robert Jurjevic]

The grades were stretched, which isn't desirable, but having gone to a lot of effort, it seems all that can be done is reduce the stretching.

The main point here is that some people believe (including me) that the starching is caused by a flaw in the current grading system and, unless the system is corrected, the stretching will continue in the future, which leaves you with two options: either live with the grades which drift more and more apart as the seasons pass or make corrections (like this one) every few seasons (say my grade jumping from 90 to 120sh).

I see no reason why one would not wish to correct the current grading system, especially when there are proposals not only to deal with the grade starching problem, but also to make it even better than the FIDE sytem (Glickman improvement implemented in Glicko 1).

Thanks a lot.

[Roger de Coverly]

Why not have the top graded players graded 1, then the next down graded 2 etc. down to about 20.

This was the way that it worked nearly sixty years ago. They still had numbers but they weren't published. Grade 1a was (I think) 248 down to 241, with 1b 240 to 233, 2a 232 to 225. Publication of the numbers and the numeric system itself took over from about the mid sixties.

If we still had the system today, then top GMs would be zero or negative because they are well above 248.

There are demands on a grading system which require greater "accuracy" than an eight point band. Ordering players for seeded swisses is but one of them.

Since you couldn't lose your grade

I don't think only being eligible for tournaments based on your peak performances would be very popular amongst older players. The USCF system (Elo based) has or had the concept of a rating floor. I believe it was a mixed blessing particularly for players whose standards had declined.

[Alex Holowczak]

The main point here is that some people believe (including me) that the starching is caused by a flaw in the current grading system and, unless the system is corrected, the stretching will continue in the future, which leaves you with two options: either live with the grades which drift more and more apart as the seasons pass or make correction (like this one) every few seasons (say my grade jumping from 90 to 120sh).

I think the average club player was happy to just let them drift apart. There are obviously a hardcore group of statisticians here who are dedicated to it, but I think 90% of the chess-playing public wouldn't have minded if things were left the way they are, i.e. with no new grade, simply because they wouldn't be aware of the problem.

I see no reason why one would not wish to correct the current grading system, especially when there are proposals not only to deal with the grade starching problem, but also to make it even better than the FIDE sytem (Glickman improvement implemented in Glicko 1).

I would love to correct the current grading system, but such is its nature, I think it is in danger of trying to follow a moving target. There are too many variables to ever make it truly reliable, e.g. ungraded players and estimating their strength, deliberate underperformance when in a team situation, juniors not playing enough games in one season and ending up counting games from three years ago, when their grade would have been much lower. Another anomoly: an ungraded player could quite easily win a tournament and play 6 games all season. He doesn't get a grade, yet everyone else's goes down, because they've lost. That'll happen an awful lot with juniors. I don't know how any grading system can account for such things, it will inevitably lead to either stretching or compression.

If we still had the system today, then top GMs would be zero or negative because they are well above 248.

Well, you just set the peak at 1 for a GM. The highest "national" grade you could get was 4, you could only get higher by entering FIDE rated tournaments (have that as the norm criteria).

There are demands on a grading system which require greater "accuracy" than an eight point band. Ordering players for seeded swisses is but one of them.

Well, the notion of an "8 point band" wouldn't exist. It'd be purely norm based. I don't think people graded 10 points apart are that much different in terms of strength. If the tournament was for grades 6-10, then you have 5 seeded groups, and just draw randomly from there. The draw would be just as valid.

I don't think only being eligible for tournaments based on your peak performances would be very popular amongst older players.

Well, everyone will be based on their peak performances, so everyone would be in the same boat. I don't think age is that much of a barrier. Look at Korchnoi, for instance.