GRADING ANOMALIES

[Alex Holowczak]

To be fair, wikipedia is very good for giving you a jist of what's going on.

[Ian Thompson]

My issue is that the system of "adding 50 for a win and subtracting 50 for a loss" is a function that has no statistical basis and has to be fudged when the difference between the opponents is more than 40, making it not properly commutative.

So its no worse than FIDE's implementation of the Elo system whose "rating scale is an arbitrary one with a class interval set at 200 points" and where "difference[s] in rating of more than 400 points shall be counted for rating purposes as though [they] were a difference of 400 points". (http://www.fide.com/fide/handbook?id=73&view=article)

[Roger de Coverly]

re Harkness system

The British grading system was first devised by civil servant Sir Richard Clarke in the late forties and early fifties. As he was a contemporary of Harkness and Elo, it's more than possible that they shared ideas about how to develop a rating system. The Harkness system has the flaw that you can lose points if you play a weak field ( or gain points if you play a strong field) no matter how badly or well you do. In its first 20 years or so, Clarke adopted the practical expedient of ignoring results with players more than 50 points apart when the result went the "right" way. With the growth of weekend tournaments, such mismatched games became more common and so the 40 point rule was invented as a means of giving the winner some credit. This was during my chess lifetime about 40 years ago.

Mike Gunn posted a long article by Clarke regarding rating systems. It's here http://www.ecforum.org.uk/viewtopic.php ... arke#p1454

The article starts

"The B.C.F. Grading System
The B.C.F. scheme followed Harkness. As soon as news of this plan reached here, the B.C.F. Development Committee began to consider the possibility of a grading system: we did not then know of the existence of the Ingo-system. I had published an article in the April, 1953, "B.C.M." on the grading of grandmasters; so then a sub-committee of Gilchrist, Buckle, Wade, and myself was set up, reported in September, 1953, and led to the first B.C.F. Grading List in March, 1954.
We adopted the Harkness notation and the basic Ingo/Harkness principle. But our situation was different, for play in this country is mainly in team events, and not in tourna¬ments. So instead of treating the tournament as a unit, like Ingo and Harkness, our unit had to be the individual game. We had to aggregate every individual's play, and calculate the average grading number of his opponents, and apply to this the premium or discount according to the difference between his percentage score and 50 per cent. Our system must reckon performance over a specified period, for players' club and county results are in units of the season's play. So we calculate and publish the results once a year, instead of continuously from tournament to tournament. We reckon (and the Americans agree) that one needs thirty games to establish a reliable grading number. We began with a three-year grading period. We quickly reduced it to two years, as we organized the flow of material. Now we grade on a one-year period, where thirty games have been played; and if the number in the year is less, fill in the difference from the previous year's performance.

The article was written in 1969 shortly before the 40 point rule was introduced as it mentions it as a possibility.

It's more than possible to be critical of the ECF grading system. Could you give us concrete reasons why it should be replaced which are something other than assertions that it's the same as a long forgotten and long abandoned system?

[Robert Jurjevic]

re Harkness system

The British grading system was first devised by civil servant Sir Richard Clarke in the late forties and early fifties. As he was a contemporary of Harkness and Elo, it's more than possible that they shared ideas about how to develop a rating system. The Harkness system has the flaw that you can lose points if you play a weak field ( or gain points if you play a strong field) no matter how badly or well you do. In its first 20 years or so, Clarke adopted the practical expedient of ignoring results with players more than 50 points apart when the result went the "right" way. With the growth of weekend tournaments, such mismatched games became more common and so the 40 point rule was invented as a means of giving the winner some credit. This was during my chess lifetime about 40 years ago.

Mike Gunn posted a long article by Clarke regarding rating systems. It's here http://www.ecforum.org.uk/viewtopic.php ... arke#p1454

The article was written in 1969 shortly before the 40 point rule was introduced as it mentions it as a possibility.

May I remark that in terms of symbols and notions used in "Grading anomalies" article at http://www.jurjevic.org.uk/chess/grade/ ... malies.htm adopting the 40 point rule is in fact switching from blue to green line in figure 1 below (green line is better than blue line as for the grade differences 'd' occurring in practice it approximates the yellow line better, yellow line is currently regarded as the best fit for the observed data).


Figure 2: Relationship between expected performance 'p' and grade difference 'd' as defined in GS and AGS3 (green line), CGS, AGS and AGS2 (blue line), ÉGS, ÉGS2, ÉGS3 and ÉGS4 (red line), ÉGS5 and ÉGS6 (yellow line), and (normal relationship 'p = 100*(1 + Erf[d/50])/2', where the error function Erf[z] is the integral of the Gaussian distribution) as originally defined by Élo (brown line above yellow). Expected performance 'p' is a function of grade difference 'd', i.e., 'p = f(d)'. Note that both FIDE and USCF switched from normal (brown line) to logistic (yellow line) relationship 'p = f(d)' which they found provides a better fit for the actual results achieved.

Let 'a' and 'b' are the grades of players 'A' and 'B', 'p' expected performance of player 'A' (expected performance of player 'B' is then '100 - p'), 'q' actual performance of player 'A' (actual performance of player 'B' is then '100 - q') and 'a2' and 'b2' new grades of players 'A' and 'B'.

'a2' and 'b2' are calculated using the following formulae (holds for any grading system mentioned here, including the current one):

Code: Select all

a2 = a + ka*(q - p);
b2 = b + kb*((100 - q) - (100 - p));

and it follows that the calculated grades 'a2' and 'b2' apart from the chosen 'p = f(d)' also depend on the chosen 'k' factors.

Could you give us concrete reasons why it should be replaced which are something other than assertions that it's the same as a long forgotten and long abandoned system?

My finding described in "Grading anomalies" article at http://www.jurjevic.org.uk/chess/grade/ ... malies.htm is that the main flaw in GS (current Grading System) is in the choice of 'k' factors (in above formulae) which are set to '1' rather than to '1/2' causing an error which accumulates over the seasons (this has nothing to do with the choice of 'p = f(d)', i.e., the 40 point rule, if you have a system with a logistic 'p = f(d)' and 'k' factors set to '1' the system would still stretch the grades, of course it is better to use logistic than linear 'p = f(d)', but having 'k' factors set to '1' rather than '1/2' affects the grade stretching much more).

In short, I think that the improvement route (in order of importance) could be as follows:
1. switch 'k' factors from '1' to '1/2' (adopting AGS3 system would do it), as this corrects the main problem here, which is grade stretching (grade stretching is why the grade corrections have been done and a new grades introduced, say my grade jumped from approximately 90 to approximately 120),
2. improve the 40 point rule by switching from green (40 points rule) to yellow (Élo logistic curve) line in figure 1 above (adopting ÉGS5 would do it), this would affect the grades of the players who played games where grade difference was grater than approximately 30 points,
3. implement (a version of) Glicko 1 (ÉGS6 implements it) and Glicko 2 (no system yet proposed by me) improvement over FIDE Élo, Glicko 1 improvement deals with grade trust regarding frequency of play (grades of players who play less frequently are less trusted and therefore changed more rapidly), Glicko 2 improvement deals with grade trust regarding grade charge (players whose grades change rapidly from season to season are less trusted and therefore changed more rapidly, this is usually referred here as "the junior problem").

[Peter Rhodes]

Could you give us concrete reasons why it should be replaced which are something other than assertions that it's the same as a long forgotten and long abandoned system?

I have already alluded to more than that Roger :

My issue is that the system of "adding 50 for a win and subtracting 50 for a loss" is a function that has no statistical basis and has to be fudged when the difference between the opponents is more than 40, making it not properly commutative.

I'm not going to put together a long post of arguments that have already been made. If you do not understand why a logarithmic function with parameters that attempt to estimate the probabilistic outcome between opponents is preferable to a flat line fudged at the edges because the function itself is of dubious value, then I am not sure you will ever understand my criticism of the system in use.

[Robert Jurjevic]

re Harkness system

The British grading system was first devised by civil servant Sir Richard Clarke in the late forties and early fifties. As he was a contemporary of Harkness and Elo, it's more than possible that they shared ideas about how to develop a rating system. The Harkness system has the flaw that you can lose points if you play a weak field ( or gain points if you play a strong field) no matter how badly or well you do. In its first 20 years or so, Clarke adopted the practical expedient of ignoring results with players more than 50 points apart when the result went the "right" way. With the growth of weekend tournaments, such mismatched games became more common and so the 40 point rule was invented as a means of giving the winner some credit. This was during my chess lifetime about 40 years ago.

Mike Gunn posted a long article by Clarke regarding rating systems. It's here http://www.ecforum.org.uk/viewtopic.php ... arke#p1454

The article was written in 1969 shortly before the 40 point rule was introduced as it mentions it as a possibility.

May I remark that in terms of symbols and notions used in "Grading anomalies" article at http://www.jurjevic.org.uk/chess/grade/ ... malies.htm adopting the 40 point rule is in fact switching from blue to green line in figure 2 below (green line is better than blue line as for the grade differences 'd' occurring in practice it approximates the yellow line better, yellow line is currently regarded as the best fit for the observed data).


Figure 2: Relationship between expected performance 'p' and grade difference 'd' as defined in GS and AGS3 (green line), CGS, AGS and AGS2 (blue line), ÉGS, ÉGS2, ÉGS3 and ÉGS4 (red line), ÉGS5 and ÉGS6 (yellow line), and (normal relationship 'p = 100*(1 + Erf[d/50])/2', where the error function Erf[z] is the integral of the Gaussian distribution) as originally defined by Élo (brown line above yellow). Expected performance 'p' is a function of grade difference 'd', i.e., 'p = f(d)'. Note that both FIDE and USCF switched from normal (brown line) to logistic (yellow line) relationship 'p = f(d)' which they found provides a better fit for the actual results achieved.

Let 'a' and 'b' are the grades of players 'A' and 'B', 'p' expected performance of player 'A' (expected performance of player 'B' is then '100 - p'), 'q' actual performance of player 'A' (actual performance of player 'B' is then '100 - q') and 'a2' and 'b2' new grades of players 'A' and 'B'.

'a2' and 'b2' are calculated using the following formulae (holds for any grading system mentioned here, including the current one):

Code: Select all

a2 = a + ka*(q - p);
b2 = b + kb*((100 - q) - (100 - p));

and it follows that the calculated grades 'a2' and 'b2' apart from the chosen 'p = f(d)' also depend on the chosen 'k' factors.

Could you give us concrete reasons why it should be replaced which are something other than assertions that it's the same as a long forgotten and long abandoned system?

My finding described in "Grading anomalies" article at http://www.jurjevic.org.uk/chess/grade/ ... malies.htm is that the main flaw in GS (current Grading System) is in the choice of 'k' factors (in above formulae) which are set to '1' rather than to '1/2' causing an error which accumulates over the seasons (this has nothing to do with the choice of 'p = f(d)', i.e., the 40 point rule, if you have a system with a logistic 'p = f(d)' and 'k' factors set to '1' the system would still stretch the grades, of course it is better to use logistic than linear 'p = f(d)', but having 'k' factors set to '1' rather than '1/2' affects the grade stretching much more).

In short, I think that the improvement route (in order of importance) could be as follows:
1. switch 'k' factors from '1' to '1/2' (adopting AGS3 system would do it), as this corrects the main problem here, which is grade stretching (grade stretching is why the grade corrections have been done and a new grades introduced, say my grade jumped from approximately 90 to approximately 120),
2. improve the 40 point rule by switching from green (40 points rule) to yellow (Élo logistic curve) line in figure 2 above (adopting ÉGS5 would do it), this would affect the grades of the players who played games where grade difference was grater than approximately 30 points,
3. implement (a version of) Glicko 1 (ÉGS6 implements it) and Glicko 2 (no system yet proposed by me) improvement over FIDE Élo, Glicko 1 improvement deals with grade trust regarding frequency of play (grades of players who play less frequently are less trusted and therefore changed more rapidly), Glicko 2 improvement deals with grade trust regarding grade charge (grades of players whose grades change rapidly from season to season are less trusted and therefore changed more rapidly, this is usually referred here as "the junior problem"), Glicko improvements allow 'ka /= kb' ('/=' means not equal, i.e. variable factors 'k' in the above formulae) but it always holds 'ka + kb = 1' (this is a necessary condition for not stretching nor shrinking the grades).

[Roger de Coverly]

I'm not going to put together a long post of arguments that have already been made. If you do not understand why a logarithmic function with parameters that attempt to estimate the probabilistic outcome between opponents is preferable to a flat line fudged at the edges because the function itself is of dubious value, then I am not sure you will ever understand my criticism of the system in use.

I think you have to use general reasoning when dealing with the advantages and disadvantages of grading systems.

Basically
(1) how does it cope with new players?
(2) how does it cope with players who play a lot or hardly at all?
(3) how does it deal with rating lag and separate that out from fluctuations in form?
(4) does give "equal" reward for equal performance?
(5) how reliable an estimate does it give that if A's grade greater than B's grade then A is a "better" chess player than B?

If you just waffle mathematical jargon "logarithmic function ... " without putting it in layman's terms , no reader of this forum including those with degrees in mathematics will have much of a clue what you are going on about.

[Robert Jurjevic]

(1) how does it cope with new players?
(2) how does it cope with players who play a lot or hardly at all?
(3) how does it deal with rating lag and separate that out from fluctuations in form?
(4) does give "equal" reward for equal performance?
(5) how reliable an estimate does it give that if A's grade greater than B's grade then A is a "better" chess player than B?

I might have the answer to some of these questions:
(1) please see "Ungraded players..." section at http://www.jurjevic.org.uk/chess/grade/ ... malies.htm (IMHO this would be the best way to deal with ungraded players)
(2) addressed in ÉGS6 (ÉGS6 is basically a simple version of Glicko 1)
(3) do not understand the question, is this sort of "junior problem"? if it is (the "junior problem") then the answer could be in a system which implements Glicko 2 idea (I haven't proposed that system yet)
(4) definitely not! please see "Equal grade for equal performance"... section at http://www.jurjevic.org.uk/chess/grade/ ... malies.htm (the reason is because it would appear that one has to choose between not giving "equal" reward for equal performance or stretching the grades, I would definitely opt for not giving "equal" reward for equal performance, i.e. if one gives "equal" reward for equal performance then one stretches the grades)
(5) that should depend on how much player grades are trusted and the grade difference, a statistician may be able to calculate that if he or she would know the game results and the method of grade calculation

Nevertheless, all these question, though fine and most of which deal with grading refinements, look to me less important until the main problem is fixed, and that is setting the sum of 'k' factors to '1' (not leaving it to '2' as it is now)! (the errors caused by the main problem are likely to be significantly greater than the errors caused by mentioned lack of fine enhancements)

[Peter Rhodes]

I think you have to use general reasoning when dealing with the advantages and disadvantages of grading systems.

Basically
(1) how does it cope with new players?
(2) how does it cope with players who play a lot or hardly at all?
(3) how does it deal with rating lag and separate that out from fluctuations in form?
(4) does give "equal" reward for equal performance?
(5) how reliable an estimate does it give that if A's grade greater than B's grade then A is a "better" chess player than B?

I've made all reasonable efforts to put forward my point, and yet the best rebuttal is a tautology ?

If I frame my case in layman's terms you ask for specific concrete examples. If I frame it in detailed terms, you explain that I am using terminology that is too complex.

You further make some comments that the casual reader will not follow the points. I have a higher regard for the intellectual capactity of ECF readers than do you, and refute what you say.

I think the only lasting impression the readers will have is of a poster trying to use all means of sophistry to avoid the central points at hand.

[Roger de Coverly]

(3) how does it deal with rating lag and separate that out from fluctuations in form?

(3) do not understand the question, is this sort of "junior problem"?

This to my mind is one of the key questions in any grading system design. How to tell the difference between a player who has "improved" by say 25 ECF points relative to his peers against a player who has just had a run of good results? I could refer you to the debate earlier this year regarding the international system and the proposed, now abandoned, doubling of the K factor.

Here's one of the many articles.

http://www.chessbase.com/newsdetail.asp?newsid=5418

As an example of the ECF system's treatment of rating lag, let's compare two players who both score 50% against an average field of 175. If they both play 30 games, then both will have new grades of 175 even if one starts with a grade of 150 and the other 175. If they only play 15 games in the grading period, then the 150 player only comes out at 163. In an Elo type system the 150 player will be heading towards 175 but the pace at which he arrives will depend on the K factor. For some combinations of number of games and K factor it's even possible to overshoot. All these systems require the player to "prove" that he's better before adjusting his strength estimate (grade) upwards to his performance.

I think the only lasting impression the readers will have is of a poster trying to use all means of sophistry to avoid the central points at hand.

What is your central point? Is it that the ECF system should be abandoned because it's similar to a long obsolete US system known as the Harkness system?

[Peter Rhodes]

Roger,

yes it is my contention that it should be abandoned.However, obviously not overnight.

It is not beyond the wit of man to store the last 5 years data-set in a database, and to proceed with a project that investigates different options and their suitabilty. All of those options would be based around the Elo system of course.

Ideally I would like to see automated grade entry, and perhaps even a "live rating" system.

[Roger de Coverly]

It is not beyond the wit of man to store the last 5 years data-set in a database, and to proceed with a project that investigates different options and their suitabilty. All of those options would be based around the Elo system of course.

The complete results history already exists and was used for the recent revaluation exercise. I would argue that the recent changes have broken the ECF's system and that the historical continuity case for retaining the Clarke system no longer exists. I doubt though that the current grading team will have much enthusiasm for another grading system rebuild.

A key point for switching to an Elo type system is that it facilitates more frequent reporting of grade updates. One advantage is that inactive players don't have grade changes which a Clarke system will sometimes do - For players playing relatively few rapidplay games, the 6 monthly list can show a dramatic change even where no games have been played in the current period.

I don't think changing the underlying assumed mathematical distribution from linear to near-linear has much practical influence. In fact I suspect that Elo type calculations running from -320 to + 320 on a linear scale would not produce any great difference in rating from those using the current international tables.


The more worrying point is that the relationship between different strengths of players may not be linear at all. Elo tables try to predict that if A is 100 Elo points above B who is 100 points above C, then A should score 62% against B who should score 62% against C. Also it tries to predict that A scores about 75% against C. But suppose the A v B and B v C predictions are correct but not the A v C.

Ideally I would like to see automated grade entry, and perhaps even a "live rating" system.

What is automated grade entry? If you have electronic collection of results, data extracts for grading are a fairly straightforward extension. Hint - always capture the ECF grading code alongside the player name.

On-line Servers ( ICC, FICS etc.) have always used "live rating" - I don't think it's practical for over the board chess.

[Peter Rhodes]

On-line Servers ( ICC, FICS etc.) have always used "live rating" - I don't think it's practical for over the board chess.

Possibly it's impractical. Possibly it's not.

The popularity of live rating sites shows that there is interest in it. People like to have immediate feedback of their performance.

[Robert Jurjevic]

This to my mind is one of the key questions in any grading system design. How to tell the difference between a player who has "improved" by say 25 ECF points relative to his peers against a player who has just had a run of good results? I could refer you to the debate earlier this year regarding the international system and the proposed, now abandoned, doubling of the K factor.

Here's one of the many articles.

http://www.chessbase.com/newsdetail.asp?newsid=5418

As an example of the ECF system's treatment of rating lag, let's compare two players who both score 50% against an average field of 175. If they both play 30 games, then both will have new grades of 175 even if one starts with a grade of 150 and the other 175. If they only play 15 games in the grading period, then the 150 player only comes out at 163. In an Elo type system the 150 player will be heading towards 175 but the pace at which he arrives will depend on the K factor. For some combinations of number of games and K factor it's even possible to overshoot. All these systems require the player to "prove" that he's better before adjusting his strength estimate (grade) upwards to his performance.

Thank you for the clarification on the grade/ rating lag.

I would assume the more (but not too many) games you take into account before changing one's grade the better, therefore the rating/ grade lag may be more of the problem to FIDE (rating after every tournament) than ECF (grading once a year).

The G30 rule

Rule G30: The Grade is calculated by dividing the total number of points scored by the number of games played. If there are at least 30 games in the current period, then the Grade is based on these games alone. If there are not, results are brought forward from the previous period to make the total up to exactly thirty. If there are not 30 games in the two seasons together, results are taken from the season before that. Games are never taken from further back than this; the maximum is two prior grading periods.

is in my opinion a good idea and should ensure (to some extent) that the grades being calcualted can be (more or less) trusted.

I assume that the G30 rule would have been applied to AGS3, ÉGS5 or ÉGS6, if one of the systems was adopted as a new ECF grading system, though omitting it for ÉGS6 would not be so critical (as the system does address the grade lag problem to some extent).

I could refer you to the debate earlier this year regarding the international system and the proposed, now abandoned, doubling of the K factor.

One should be careful with ECF 'k' factors either for Élo/ logistic or linear ECF grading systems, i.e., ECF 'ka' and 'kb' factors can be changed but always in such a way that 'ka + 'kb' = 1', otherwise the ECF system would stretch or shrink the grades, sadly the current ECF grading system has 'ka + kb = 1 + 1 = 2', this should also apply to FIDE 'k' factors, as basically both FIDE and ECF use the same general formula (what is different is grade/ rating scale, frequency of grading/ rating, the choice of 'k' factors and the relationship between expected performance 'p' and grade/ rating difference 'd').

[Robert Jurjevic]

On-line Servers ( ICC, FICS etc.) have always used "live rating" - I don't think it's practical for over the board chess.

FICS is using Glicko 1 (BTW I'm running SurveyBot there).