GRADING ANOMALIES

[Brian Valentine]

[quote][/I don't think changing the underlying assumed mathematical distribution from linear to near-linear has much practical influence. In fact I suspect that Elo type calculations running from -320 to + 320 on a linear scale would not produce any great difference in rating from those using the current international tables.quote]

Roger,
I can help you out a bit here: Elo in The rating of chessplayers past and present says this in 1.81: "Examination of the percentage expectancy curve... shows that between -1.5C and 1.5 C (+/300 Elo, +/37 ECF)...... [the curve]... may be approximated by a straight line". So support from "the bible".

Reading the thread written this month, I am struggling to see the new ideas that change the situation from the torpur the thread had reached. In a sense in the majority of situations the systems are equivalent. The ECF system has been adapted to meet the English fixture organisation and resources put into grading. To my eyes the ECF system is discredited by the changes effective this summer and this gives grounds for long term review. However the debate should not be about linear approximations to log curves, but how to deal with changes in performance (real changes in strength against fluctuations) and the deflation/ inflation due to turnover of players. Robert's perceived natural stretch is also a red herring.

[Robert Jurjevic]

Robert's perceived natural stretch is also a red herring.

My understanding is that this thread is about the need to correct the anomalies in the current grading system, not only correcting the accumulated error now and than by hand introducing new grades, but rather, if possible, to correct the calculation method itself, so that the error does not accumulate any more, in other words, to propose the system which would not stretch the grades.

I believe that the stretch is not 'natural', and I would wish to calculate and publish AGS3 grades for 2009, 2010, etc. (I can calculate AGS3 grades without having actual game results, all I need is GS grades for 2008, 2009, 2010, etc., which were or will be published), which should allow for a comparison of the two grade sets.

According to GS I was 90ish, pretty much constant over about five seasons, in 2009 I became 120ish (new corrected grade), if my chess abilities remain relatively constant I expect to become 90ish again after a couple of seasons (do not know how many) but hope to stay 120ish according to AGS3.

[Ola Winfridsson]

(3) how does it deal with rating lag and separate that out from fluctuations in form?

(3) do not understand the question, is this sort of "junior problem"?

This to my mind is one of the key questions in any grading system design. How to tell the difference between a player who has "improved" by say 25 ECF points relative to his peers against a player who has just had a run of good results? I could refer you to the debate earlier this year regarding the international system and the proposed, now abandoned, doubling of the K factor.

I apologize in advance for asking a daft question, but how can you tell? As far as I can see the only way to ensure this would be to publish the grading list with a very high frequency or use a live rating system. Secondly, should you differentiate between them? Surely it's just as important that a good run of results (or perhaps we should call it "good form") is reflected in the grade as a more permanent improvement? Otherwise, the grading system wouldn't give a reasonably accurate indication of the results to be expected.

[Brian Valentine]

Ola,
Yours is a good question. The system, after a result, can at one extreme not move at all and at the other move to the grade achieved in this one game. Neither extreme provides a useful answer and hence the debate is about what the "k" and 1-k to apply to each extreme to get a useful compromise.

This is a fairly trivial exercise if the starting grades are sensible, there is no turnover of players and performance is stable. The problems arise if any one of these assumptions is false.

[Roger de Coverly]

I apologize in advance for asking a daft question, but how can you tell?

One of the problems is that you cannot. You can infer that junior change is the result of improvement but never be absolutely certain. The ECF grading team has heeded the plea that junior grades are usually out of date at the cost of removing stability and continuity from one year to the next. FIDE had a similar issue with doubling the K factor and opted for stability instead of accelerating the revaluation of improving players.

if my chess abilities remain relatively constant I expect to become 90ish again after a couple of seasons (do not know how many)

I rather expect that you (and your peers) will stay at about 120, but you will find that Mark (Hebden) and Keith (Arkell) become 10 to 20 points higher than they are now. I don't think you can expect to shift the mean and median of a grading list 25 points higher and reduce the distance between top and middle without the increases sticking to players in the middle and inflating players at the top.

[Robert Jurjevic]

if my chess abilities remain relatively constant I expect to become 90ish again after a couple of seasons (do not know how many)

I rather expect that you (and your peers) will stay at about 120, but you will find that Mark (Hebden) and Keith (Arkell) become 10 to 20 points higher than they are now. I don't think you can expect to shift the mean and median of a grading list 25 points higher and reduce the distance between top and middle without the increases sticking to players in the middle and inflating players at the top.

I think you are right, I admit that I made a mistake assuming that my GS grade of 120 would drop to 90 if I perform as expected at the level of 120, it should stay at 120 as you have assumed.

Nevertheless, I still think that GS stretches the grades. GS awards (penalizes) players who over-perform (under-perform) approximately (if there was not for the G30 rule it would be exactly) double than AGS3 would. Therefore (if we keep calculating both GS and AGS3 grades) GS and AGS3 grades should drift apart for the players who under-performed (over-performed). Say, you have a promising junior who over-performs from season to season, after couple of seasons his GS grade would be higher than his AGS3 grade (for the same performance), likewise can be said for a player who under-performs, his GS grade would be lower than his AGS3 grade (grade differences between over-performers and under-performers are bigger on average in GS than AGS3, or GS grades are stretched in respect to AGS3 grades). I think that there should be not doubt that GS and AGS3 grades would diverge if there are players who over-performed (under-performed) on average (it is enough to look at their formulae to come to that conclusion). I claim that AGS (and not GS) applies the right amount of grade correction ('ka = kb = 1/2' and not 'ka = kb = 1') with which you may not agree (but I tried to explain why I think so in the "Grading anomalies" article at http://www.jurjevic.org.uk/chess/grade/ ... malies.htm ). Thanks.

[Roger de Coverly]

Nevertheless, I still think that GS stretches the grades

The ECF system is equivalent to calculating a new grade (for a season) as (old grade)*(1-factor) + (factor)*performance. "factor" is 100% for 30 or more games and 0% for zero games. It varies between 0% and 100% for players with less than 30 games in the current period depending on the number of games in previous periods. In practice it's a bit more complicated where more than one season is involved and the "old grade" was also over more than one season.

I believe you advocate setting "factor" at 50% regardless of the number of games. So this will reduce lag for players with less than 15 games a season and increase lag for those with more than 15 games.

As regards stretching, I would observe that the mean and median of the grading system showed no great movement (looking back to 1994) and neither did the grades of the top players. In fact if players were underperforming, then the grading range should have compressed until equilibrium was reached. As the grading range (from middle to top) remained roughly stable for many years even when 25 points difference was scoring 69%, my conclusion is that there are equal and opposite forces acting. The 40 point rule is usually a bit generous to the higher graded player so that's one possible suspect. Non-linearity of the underlying strength distribution is another (so that 150 scores 40% against 160 who scores 40% against 170 but 150 doesn't score 30% against 170).

[Robert Jurjevic]

The ECF system is equivalent to calculating a new grade (for a season) as (old grade)*(1-factor) + (factor)*performance. "factor" is 100% for 30 or more games and 0% for zero games. It varies between 0% and 100% for players with less than 30 games in the current period depending on the number of games in previous periods. In practice it's a bit more complicated where more than one season is involved and the "old grade" was also over more than one season.

I believe you advocate setting "factor" at 50% regardless of the number of games. So this will reduce lag for players with less than 15 games a season and increase lag for those with more than 15 games.

No, the suggestions in all of the systems were aimed towards forumlae (or rules) which are used in calculation of grade corrections for individual games and the G30 rule is assumed to be applied for all AGS3, ÉGS5 and ÉGS6, with a possibility of its exclusion for ÉGS6, as ÉGS6 estimates the trust level of the grades based on the frequency of play and changes less trusted grades more rapidly than more trusted grades (so having grades calculated based on fewer games would not be an issue).

Rule G30: The Grade is calculated by dividing the total number of points scored by the number of games played. If there are at least 30 games in the current period, then the Grade is based on these games alone. If there are not, results are brought forward from the previous period to make the total up to exactly thirty. If there are not 30 games in the two seasons together, results are taken from the season before that. Games are never taken from further back than this; the maximum is two prior grading periods.

Switching to AGS3 would be as simple as replacing the existing rule

Rule 1a: For a win you score your opponent's grade plus 50; for a draw, your opponent's grade; and for a loss, your opponent's grade minus 50. Note that, if your opponent's grade differs from yours by more than 40 points, it is taken to be exactly not 40 points above (or below) yours. At the end of the season an average of points-per-game is taken, and that is your new grade.

with

Rule 2b: For a win you score average grade plus 25; for a draw, average grade; and for a loss, average grade minus 25. Average grade is half of the sum of your and your opponent's grade. Note that, if your opponent's grade differs from yours by more than 40 points, it is taken to be exactly 40 points above (or below) yours. At the end of the season an average of points-per-game is taken, and that is your new grade.

As regards stretching, I would observe that the mean and median of the grading system showed no great movement (looking back to 1994) and neither did the grades of the top players. In fact if players were under-performing, then the grading range should have compressed until equilibrium was reached. As the grading range (from middle to top) remained roughly stable for many years even when 25 points difference was scoring 69%, my conclusion is that there are equal and opposite forces acting. The 40 point rule is usually a bit generous to the higher graded player so that's one possible suspect. Non-linearity of the underlying strength distribution is another (so that 150 scores 40% against 160 who scores 40% against 170 but 150 doesn't score 30% against 170).

If there is not for the G30 rule both GS (current Grading System) and AGS3 (Amended Grading System three) would preserve the total grade of the system and consequently (assuming that the effect of the G30 rule is relatively small) one would expect the mean to remain fairly constant over the seasons, and if the distribution is approximately symmetrical around the mean, median should be approximately equal to the mean, thus remaining fairly constant too.

I think that the grade stretching did not occur because the players were under-performing on average (it is more likely that they were over-performing), but rather because under-performing players were (and still are) penalized and over-performing players were (and still are) rewarded too many grading points (which is now known to be roughly twice as many than it should have been) and the grade gap between under-performers and others and over-performers and others kept increasing beyond the level at which it should have been. In spite of all this going on (for the system which approximately preserves the total system grade) the mean and the median (if the distribution kept symmetrical around the mean) may have remained roughly constant.

[Roger de Coverly]

Rule 2b: For a win you score average grade plus 25; for a draw, average grade; and for a loss, average grade minus 25. Average grade is half of the sum of your and your opponent's grade.

So under your system, your new grade is 50% of your old grade and 50% of your performance. So a player graded 125 scoring 50% against opposition averaging 150 would get a new grade of 138, whereas a player graded 120 would get a new grade of 135 with the same performance. This is the same as under the ECF system in the special case where only 15 games have been played.

Mind you I don't think you accept that scoring 50% against a 150 field is a 150 performance independent of the grade of the person doing the performing.

[Robert Jurjevic]

So under your system, your new grade is 50% of your old grade and 50% of your performance. So a player graded 125 scoring 50% against opposition averaging 150 would get a new grade of 138, whereas a player graded 120 would get a new grade of 135 with the same performance. This is the same as under the ECF system in the special case where only 15 games have been played.

Yes, though, if really 15 games were played, the AGS3 grades would not be 138 (137.5) and 135, but the same correction would have been applied taking into account performance in previous season (or seasons), as it would have been done for GS (current Grading System) grades.

Mind you I don't think you accept that scoring 50% against a 150 field is a 150 performance independent of the grade of the person doing the performing.

No, in principle I wouldn't, though it is possible to assume that the player did perform as 150 player and be fully awarded, but then it has to be assumed that the pool players also performed at 150 level and should not be penalized for the games they have played against the player. Why I think that "equal grade for equal performance" rule has to be abandoned is described in "Equal grade for equal performance..." section at http://www.jurjevic.org.uk/chess/grade/ ... malies.htm (if one accepts "equal grade for equal performance" rule one automatically stretches the grades, and that is what we want to avoid).

[Robert Jurjevic]

AGS3 grades for 2009...

I have calculated AGS3 grades for 2009 (2008/2009 season) using GS grades for 2009 published in gradeslive.csv document which I downloaded at... http://grading.bcfservices.org.uk/downloads.php (I assumed that 2008 EGS3 grades are equal to 2008 GS grades and that number of games on which 2008 GS and AGS3 grades are based is 0).

The G30 rule below has been taken into account in the calculation.

Rule G30: The Grade is calculated by dividing the total number of points scored by the number of games played. If there are at least 30 games in the current period, then the Grade is based on these games alone. If there are not, results are brought forward from the previous period to make the total up to exactly thirty. If there are not 30 games in the two seasons together, results are taken from the season before that. Games are never taken from further back than this; the maximum is two prior grading periods.

The results can be found in ags3grade09.zip file which can be downloaded from... http://www.jurjevic.org.uk/chess/grade/. The (zip) file contains ags3grade09.txt (textual tab delimited), ags3grade09.xls (Excel spreadsheet document) and gradeslive.csv (Excel spreadsheet document).

The calculation was performed by a Windows .net console (command line) application (written in C# programming language) interfacing with Oracle Database 10g Express Edition database. Histograms are obtained with Mathematica 7.


Figure 1: Histogram for 2009 GS grades (consisting of 10153 grades in bins [0,1), [1,2), etc., of 1 grading point width, where minimum grade is 0.00, median grade is 133.00, maximum grade is 281.00, mean grade is 133.69 and standard deviation is 37.05).


Figure 2: Histogram for 2009 EGS3 grades (consisting of 10153 grades in bins [0,1), [1,2), etc., of 1 grading point width, where minimum grade is 0.00, median grade is 133.50, maximum grade is 281.00, mean grade is 133.55 and standard deviation is 36.73).

[Roger de Coverly]

wrote:
Mind you I don't think you accept that scoring 50% against a 150 field is a 150 performance independent of the grade of the person doing the performing.

No, in principle I wouldn't,

I think you are saying that if you take two players who play exactly the same opponents with exactly the same results, then their performances are not the same. Common sense says they are and this is independent of the design of the ranking, grading or rating system. I say performance rather than grade or rating because ranking, grading or rating systems weight past rating and current performance to get current rating. For a game count greater than 30, the ECF system gives zero weight to past grade, whereas in an Elo system, the K factor determines the game count.

[Robert Jurjevic]

I think you are saying that if you take two players who play exactly the same opponents with exactly the same results, then their performances are not the same. Common sense says they are and this is independent of the design of the ranking, grading or rating system. I say performance rather than grade or rating because ranking, grading or rating systems weight past rating and current performance to get current rating. For a game count greater than 30, the ECF system gives zero weight to past grade, whereas in an Elo system, the K factor determines the game count.

I think that the performances of the two players who played the same opponents with exactly the same results are not necessarily identical as their opponents may have not played at the exactly same level in both of the rounds.

If one would know the grades, one could calculate if the players performed better, worse or as expected, and if they did perform better or worse this may be because either the pool players, them or all played below or above their level, IMHO one should not assume that on average the pool players performed as expected (the fact that they belong to a pool should not give them any special status as they are also regarded as individual players respectively to the pools of players they have played).

Let us assume that Roger 140 and Robert 120 play 30 games both scoring 50%. If one would want to calculate Roger's and Robert's new grade based on the 30 games they have played then it should be plausible to expect that their new grades should be equal.

IMHO there are three valid principal cases here: 1) Roger's new grade is 140 and Robert's new grade is 140 (both Roger and Robert played at 140 level), 2) Roger's new grade is 120 and Robert's new grade is 120 (both Roger and Robert played on 120 level) and 3) Roger's new grade is 130 and Robert's new grade is 130 (both Roger and Robert played at 130 level). (note that cases in which new grades of say ..., 126, 127, 128, 129, 131, 132, 133, 134,... were assigned to both players would also be valid)

According to GS (current Grading System) Roger's new grade would be 120 and Robert's new grade would be 140, which IMHO makes no sense as Roger's new grade is not equal to Robert's new grade, the difference is 20 grading points, i.e. GS stretched in this example the grades of Roger and Robert relatively to each other for 20 grading points.

According to AGS3 Roger's new grade would be 130 and Robert's new grade would be 130 (both Roger and Robert played at 130 level), that is because AGS3 assumes that the reason why the result was not as expected is because both Roger did play below his level and Robert did play above his level (splitting Roger's under- and Robert's over- performance 50/50).

If say Robert was ungraded then according to ÉGS6 Roger's new grade would be 140 and Robert's new grade would be 140 (it would be assumed that both Roger and Robert played at 140 level as Robert's grade would not be trusted at all relatively to Roger's grade).

If instead of Roger Robert played a pool of players (where each of the players was) graded 140 scoring 50% (assuming all draws) it would be possible to assume that each pool player did play at 140 level (though I see no obvious reason for such an assumption) and that Robert's new grade is 140, but then, when calculating the grades of the pool players, one should not penalize them for their draws against Robert (like those games are excluded from grading of the pool players) as it is already assumed that they have played at 140 level in those games.

[Roger de Coverly]

I think that the performances of the two players who played the same opponents with exactly the same results are not necessarily identical as their opponents may have not played at the exactly same level in both of the rounds.

According to Elo, a grade or a rating represents an estimate of an unknown random variable called chess strength. The only information you have on which to base ratings is results of games played. You do not have access to subjective information about how well or badly a player performed in a particular game. If you did, or if you could measure the accuracy of play in individual games then you could possibly come up with a grading or rating system that also reflected quality of play. In its absence, rating theory could only ever use results.

Let us assume that Roger 140 and Robert 120 play 30 games both scoring 50%.

I expect they are lighthouse keepers as well. We'll stop though after 15 games and publish a new grading list. If we do that then they are both 130 on the new list and will both be 130 while they continue to draw their match. They are correctly relatively ranked as equal. Whether the absolute value of 130 is correct relative to the rest of the chess playing population won't be known unless they get out a bit more and play them.

one should not penalize them

I don't like this terminology - it isn't a question of penalties more a question of putting players into a ranking order which has the extra property of providing an insight into the more likely results of a series of games.

[Robert Jurjevic]

According to Elo, a grade or a rating represents an estimate of an unknown random variable called chess strength. The only information you have on which to base ratings is results of games played. You do not have access to subjective information about how well or badly a player performed in a particular game. If you did, or if you could measure the accuracy of play in individual games then you could possibly come up with a grading or rating system that also reflected quality of play. In its absence, rating theory could only ever use results.

It is impossible to measure chess abilities independently of chess performances (there is not a device one can put on the heads of chess players and get a measure of their chess abilities), if that would be possible, one would be able to plot 'p' against 'd' and find the best fit for 'p = f(d)'. Nevertheless, assuming that for small differences in chess abilities (say '|d|<=30') the relationship between chess performance and difference in chess abilities is linear, one can assume that grades for '|d|<=30' are in fact chess abilities and, taking into account game records where '|d|>30', plot 'p' against 'd' (black dots in the figure 3 below) and find that 'p = f(d)' for '|d|>30' follows one of the sigmoid curves (yellow, brown and red lines in the figure 3) closer than linear approximations (green and blue lines in the figure 3). (please look at "Which 'p = f(d)'..." section at... http://www.jurjevic.org.uk/chess/grade/ ... malies.htm)

Therefore, a grade should be an estimate of chess strength (or ability) even the choice of 'p = f(d)' appears to be arbitrary (which it is not), i.e. even... the only information one has on which to base grades is the results of the games played, and even one does not have access to subjective information about how well or badly a player performed in a particular game.

If a grade is an estimate of chess strength, then, if the players did not perform as expected, their grade corrections in fact represent the correction of their estimated chess strengths, and IMHO this cannot be done without assumptions about how well or badly the players performed in their games, as (if grades are treated as real numbers) there is an infinite number of grade corrections which would fit the new game results, the ambiguity is resolved by assuming how well or badly the players performed in their games (which is in most cases to take that the player whose grade is going down weakened and the player whose grade is going up strengthen for a half of the total correction to be applied), unfortunately, GS's correction is twice as big than it should have been, i.e., correcting the grades according to GS even does not fit the new game results (too much correction is applied to both, those improved and those worsened).

Let us assume that Roger 140 and Robert 120 play 30 games both scoring 50%.

I expect they are lighthouse keepers as well. We'll stop though after 15 games and publish a new grading list. If we do that then they are both 130 on the new list and will both be 130 while they continue to draw their match. They are correctly relatively ranked as equal. Whether the absolute value of 130 is correct relative to the rest of the chess playing population won't be known unless they get out a bit more and play them.

Why would you stop after 15 games (shouldn't the same rules apply to all)? If Roger and Robert kept playing 30 or more games per season scoring 50% in every subsequent season their grade would kept oscillating between 140 and 120, which is obviously wrong, as they should have been estimated as equally strong players (GS clearly failed here).

The lighthouse keepers example is not the only one showing GS stretching the grades (below follows an another one).

Let us assume that Roger 120 and Robert 120 play 30 games Roger scoring 70% and Robert scoring 30%. If one would want to calculate Roger's and Robert's new grade based on the 30 games they have played then it should be plausible to expect that Roger's new grades should be 20 grading points greater than Robert's new grade.

IMHO there are three valid principal cases here: 1) Roger's new grade is 140 and Robert's new grade is 120 (Roger played at 140 level and Robert played at 120 level), 2) Roger's new grade is 120 and Robert's new grade is 100 (Roger played at 120 level and Robert played at 100 level) and 3) Roger's new grade is 130 and Robert's new grade is 110 (Roger played at 130 level and Robert played at 110 level).

According to GS (current Grading System) Roger's new grade would be 140 and Robert's new grade would be 100, which IMHO makes no sense as Roger's new grade is not 20 grading points greater than Robert's new grade, the difference is 40 grading points, i.e. GS stretched in this example the grades of Roger and Robert relatively to each other for 20 grading points.

According to AGS3 Roger's new grade would be 130 and Robert's new grade would be 110 (Roger played at 130 level and Robert played at 110 level), that is because AGS3 assumes that the reason why the result was not as expected is because both Roger did play above his level and Robert did play below his level (splitting Roger's over- and Robert's under- performance 50/50).