GRADING ANOMALIES

[Robert Jurjevic]

ÉGS6 calculation...

If I would be given game results at the end of every season (starting from 2008/2009) I should be able to calculate ÉGS6 grades for all players.

Then by comparing GS and ÉGS6 grade sets one can decide which of the two systems behaves better (if any) in respect to grade stretching (probably by performing after a few years the same statistical analysis which has lead to the recent grade correction).

Data needed...

I would need game result data in some suitable form, say as .csv files (if required, I may be able to provide some help with game data extraction from ECF grading database, say if the database is an SQL relational database, after finding names of relevant tables and columns, the game data should be possible to extract by issuing a series of SQL commands)

The player info I could get from published .csv files in download section... http://grading.bcfservices.org.uk/downloads.php (this would be needed to initially populate my database with relevant player info and keep it up to date, say I can use a 2009 gradeslive.csv file to initially populate the database and then 2010, 2011, etc. gradeslive.csv files to keep it up to date, adding new players)

Computer program...

It should not be difficult for me to write a computer program which would perform these calculations. This would most likely be a simple Windows .net console (command line) application written in C# programming language operating on Oracle Database 10g Express Edition database.

Data provided

I could present the results in a form of .csv files (say egs6grades09.csv, egs6grades10.csv, etc.)

"Grading anomalies" document (19/11/2009 2.2)...

Please note that a detailed discussion on the mentioned grading systems can be found in updated "Grading anomalies" document (19/11/2009 2.2) at... http://www.jurjevic.org.uk/chess/grade/ ... malies.htm

Thanks.

[Robert Jurjevic]

AGS3 grade calculation using GS grades...

As the difference between GS and AGS3 formulae is only in 'k' factors (GS's 'k' factors are 1 and AGS3's are 1/2) one can calculate AGS3 grades either using game results (as one would normally do) or using GS grades calculated using the game results (not using the actual game results directly).

Though ÉGS5 and ÉGS6 should give better results (ÉGS5 in games with larger grade difference, and ÉGS6 both in games with larger grade difference and in games where one of the players is inactive or ungraded) AGS3 should deal with the main problem referred here as "the grade stretching" approximately equally good.

AGS3 grades for 2009...

I have calculated AGS3 grades for 2009 (2008/2009 season) using GS grades for 2009 published in gradeslive.csv document which can be downloaded at... http://grading.bcfservices.org.uk/downloads.php (I assumed that 2008 EGS3 grades are equal to 2008 GS grades and that number of games on which 2008 GS and AGS3 grades are bases is 0, in 2008 new corrected grades were calculated in parallel with the old grades for the first time though the old grades were officially used but for the last time).

The G30 rule below has been taken into account in the calculation.

Rule G30: The Grade is calculated by dividing the total number of points scored by the number of games played. If there are at least 30 games in the current period, then the Grade is based on these games alone. If there are not, results are brought forward from the previous period to make the total up to exactly thirty. If there are not 30 games in the two seasons together, results are taken from the season before that. Games are never taken from further back than this; the maximum is two prior grading periods.

!
The results are in ags3grade09.zip file which can be downloaded from... http://www.jurjevic.org.uk/chess/grade/! The (zip) file contains ags3grade09.txt (textual tab delimited) and ags3grade09.xls (Excel spreadsheet document) files (there are 11,701 players of which for 10,205 AGS3 grade was calculated)!

I should be able to calculate AGS3 grades for 2010 soon after 2010 GS grades are published. I expect the two grade sets to diverge as the seasons pass (GS grades 'stretching' relatively to AGS3 grades).

Computer program...

The calculation was performed by a Windows .net console (command line) application (written in C# programming language) interfacing with Oracle Database 10g Express Edition database.

AGS3 grades calculation details...

AGS3 grades for years 2009, 2010, 2011, etc., for each player (who played 30 or more games in each season) can be calculated as follows:

Code: Select all

AGS3_08 = GS_08
AGS3_09 = AGS3_08 + (GS_09 - GS_08) / 2 
AGS3_10 = AGS3_09 + (GS_10 - GS_09) / 2 
AGS3_11 = AGS3_10 + (GS_11 - GS_10) / 2 
etc.

where 08 indicates year 2008 (or season 2007/2008), 09 year 2009 (or season 2008/2009), etc., AGS3 denotes an AGS3 type grade and GS a GS (current Grading System) type grade.

In order to take into account the above rule G30 (mentioned in "AGS3 grades for 2009..." section) one has to use a bit different formulae and the calculation algorithm may look like (C# programming language):

Code: Select all

int n0;
int n1;
int n2;
int n3;
double GS0;
double GS1;
double GS2;
double GS3;
double AGS0;
double AGS1;
double AGS2;
double AGS3;
// read data from database
int n31 = 0;
int n32 = 0;
int n33 = 0;
int n20 = 0;
int n21 = 0;
int n22 = 0;
double gs2 = 0;
double gs3 = 0;
double ags3 = 0;
if (n3 >= 30)
{
  n31 = n32 = 0;
  n33 = n3;
}
else 
{
  if (n2 + n3 >= 30)
  {
    n31 = 0;
    n32 = 30 - n3;
    n33 = n3;
  }
  else
  {
    if (n1 + n2 + n3 >= 30)
    {
      n31 = 30 - n3 - n2;
      n32 = n2;
      n33 = n3;
    }
    else
    {
      n31 = n1;
      n32 = n2;
      n33 = n3;
    }
  }
}
if (n2 >= 30)
{
  n20 = n21 = 0;
  n22 = n2;
}
else 
{
  if (n1 + n2 >= 30)
  {
    n20 = 0;
    n21 = 30 - n2;
        n22 = n2;
  }
  else
  {
    if (n0 + n1 + n2 >= 30)
    {
      n20 = 30 - n2 - n1;
      n21 = n1;
      n22 = n2;
    }
    else
    {
      n20 = n0;
      n21 = n1;
      n22 = n2;
    }
  }
}
gs3 = GS3;
if (n33 > 0)
{
    gs3 = (-(GS1 * n31) - GS2 * n32 + GS3 * (n31 + n32 + n33)) / n33;
}
gs2 = GS2;
if (n22 > 0)
{
    gs2 = (-(GS0 * n20) - GS1 * n21 + GS2 * (n20 + n21 + n22)) / n22;
}
ags3 = GS3;
if (s > 1)
{
    ags3 = gs2 + (gs3 - gs2) / 2;
}
AGS3 = (n31*AGS1 + n32*AGS2 + n33*ags3)/(n31 + n32 + n33);

where index 3 denotes current season (for which we are calculating AGS3 grade) and indices less than 3 previous seasons, 's' is a number of seasons with available grading data (we may need data from up to 4 seasons), 'n' denotes a number of games played (in a season), GS denotes a GS (current Grading System) type grade, AGS denotes an AGS3 type grade (note that AGS3 is an AGS3 grade in season 3 and AGS2 an AGS3 grade in season 2, etc.), 'gs' and 'ags' are grade performances (for a season) needed in calculation, 'n' with two indices represent various number of games needed in calculation.

"Grading anomalies" document (19/11/2009 2.2)...

Please note that a detailed discussion on the mentioned grading systems can be found in updated "Grading anomalies" document (19/11/2009 2.2) at... http://www.jurjevic.org.uk/chess/grade/ ... malies.htm

Thanks.

[Peter Rhodes]

The biggest anomaly I can see here is the Harkness system in a world that uses Elo's method.

I had no idea that people were investing such time into Harkness. I find the entire effort perplexing at best.

[Alex Holowczak]

I just downloaded the .xls file (couldn't be bothered with the hassle of getting the .exe to run on Wine...), but your ASG3 seems to do the trick. The inflated juniors seemed about right relative to adults, while still remaining in the right sort of order. Kilpatrick is set at 214, which makes a bit more sense.

My grade was still shockingly high though.

[Robert Jurjevic]

The biggest anomaly I can see here is the Harkness system in a world that uses Elo's method.
I had no idea that people were investing such time into Harkness. I find the entire effort perplexing at best.

Although CS (current Grading System) and AGS3 may have some similarities with Harkness' system, namely linear approximation between expected performance 'p' and grade difference 'd' (please see green line in figure 1 at http://www.jurjevic.org.uk/chess/grade/ ... malies.htm ), both ÉGS6 and ÉGS5 use the same logistic relationship between expected performance 'p' and grade difference 'd' (please see yellow line in figure 2 at http://www.jurjevic.org.uk/chess/grade/ ... malies.htm ) as FIDE Élo (ÉGS6 should be even better than current FIDE Élo system).

Why to me ECF grade scale makes more sense than FIDE rating scale is described in "ECF grade vs FIDE rating scale..." section at http://www.jurjevic.org.uk/chess/grade/ ... malies.htm .

AGS3 should not stretch the grades.

[Paul McKeown]

The biggest anomaly I can see here is the Harkness system in a world that uses Elo's method.

I had no idea that people were investing such time into Harkness. I find the entire effort perplexing at best.

I don't know where you get the idea that the Harkness system is in use anywhere in the world; certainly the ECF don't use it, never have. The USCF gave up in it nearly 40 years ago when they implemented Elo's system. No other organisation ever implemented it.

The ECF's system was designed by Sir Richard Clarke. Fundamentally it remains unchanged, although various fiddle factors have been introduced - and for some dubious reasons, the gnomes decided to completely bork it this year. It still retains Clarke's method, just with a new baseline.

Do you want to accuse anyone of using the Ingo system as well, for good measure?

[Peter Rhodes]

I don't know where you get the idea that the Harkness system is in use anywhere in the world; certainly the ECF don't use it, never have

Paul, allow me to help you:

When a player competes in a tournament, the average rating of his competition is calculated. If the player scores 50% he receives the average competition rating as his performance rating. If he scores more than 50% he new rating is the competition average plus 10 points for each percentage point above 50. If he scores less than 50% his new rating is the competition average minus 10 points for each percentage point below 50 (Harkness 1967:185-88).

It is fairly obvious that the current system in use by the ECF is close enough to the Harkness System as to be described as such. Whether there are some minor deviations is simply splitting hairs.

[Roger de Coverly]

It is fairly obvious that the current system in use by the ECF is close enough to the Harkness System as to be described as such. Whether there are some minor deviations is simply splitting hairs.

They are similar only in the sense that both are rating systems. The Harkness system shares withmost Elo type systems the feature that it recalculates the rating after each event, which the ECF system does not.

Example

A player with a rating of 1600 plays in an eleven-round tournament and scores 2½-8½ (22.7%) against competition with an average rating of 1850. This is 27.3% below 50%, so his new rating is 1850 - 10 × 27.3 = 1577 (Harkness 1967:187).

A Clarke style system calculates not by event but by time period, usually a year.

That said, the changes made this year have (in my opinion) damaged the ECF's system very possibly beyond repair and switching to a Elo type system should now be on the agenda.

[Robert Jurjevic]

That said, the changes made this year have (in my opinion) damaged the ECF's system very possibly beyond repair and switching to a Elo type system should now be on the agenda.

It is fairly obvious that the current system in use by the ECF is close enough to the Harkness System as to be described as such. Whether there are some minor deviations is simply splitting hairs.

My understanding is that when one compares the current ECF (here known as GS) or Harkness and FIDE grading systems they refer to the fact that the former use linear and the latter logistic relationship between expected performance 'p' and grade difference 'd'. How often one grades is a mater of choice and is independent of the chosen 'p'-'d' relationship (the more often one grades the smaller the 'k' factors should be).

If two systems differ only in choice of 'p = f(d)', say one uses a linear (green or blue line in figure 1 below) and other a logistic (red or yellow line in figure 1 below) relationship between expected performance 'p' and grade difference 'd', the difference between the two systems should be noticeable in practice only for games where grade difference is larger than approximately 30 grading points (i.e. if only games with grading difference of 30 points or less were played both systems would give fairly similar results).


Figure 1: Relationship between expected performance 'p' and grade difference 'd' as defined in GS and AGS3 (green line), CGS, AGS and AGS2 (blue line), ÉGS, ÉGS2, ÉGS3 and ÉGS4 (red line), ÉGS5 and ÉGS6 (yellow line), and (normal relationship 'p = 100*(1 + Erf[d/50])/2', where the error function Erf[z] is the integral of the Gaussian distribution) as originally defined by Élo (brown line above yellow). Expected performance 'p' is a function of grade difference 'd', i.e., 'p = f(d)'. Note that both FIDE and USCF switched from normal (brown line) to logistic (yellow line) relationship 'p = f(d)' which they found provides a better fit for the actual results achieved.

Let 'a' and 'b' are the grades of players 'A' and 'B', 'p' expected performance of player 'A' (expected performance of player 'B' is then '100 - p'), 'q' actual performance of player 'A' (actual performance of player 'B' is then '100 - q') and 'a2' and 'b2' new grades of players 'A' and 'B'.

'a2' and 'b2' are calculated using the following formulae (holds for any grading system mentioned here, including the current one):

Code: Select all

a2 = a + ka*(q - p);
b2 = b + kb*((100 - q) - (100 - p));

and it follows that the calculated grades 'a2' and 'b2' apart from the chosen 'p = f(d)' also depend on the chosen 'k' factors.

In my opinion, the main flaw in GS (current Grading System) is in the choice of 'k' factors which are set to '1' rather than to '1/2' causing an error which accumulates as the seasons pass (this has nothing to do with the choice of 'p = f(d)', if you have a system with a logistic 'p = f(d)' and 'k' factors set to '1' the system would still stretch the grades, of course it is better to use logistic than linear 'p = f(d)', but having 'k' factors set to '1' rather than '1/2' affects the grade stretching much more).

Relevant extracts from http://www.jurjevic.org.uk/chess/grade/ ... malies.htm follow...

Replacing GS...

In my opinion three candidates for replacing GS are AGS3, ÉGS5 and ÉGS6, with ÉGS6 as the best and ÉGS5 as the second best.

AGS3 is GS with the 'k' factors equal to '1/2' (the ECF's linear approximation is used for 'p = f(d)'; does not stretch the grades).

ÉGS5 is sort of ECF equivalent of FIDE's Élo (logistic curve is used for 'p = f(d)', this is regarded as more accurate than the ECF's linear approximation; the grades are ECF grades, not FIDE ratings, i.e., a strong grandmaster is about 270 not 2800; grading is done every season rather than after every tournament; does not stretch the grades).

ÉGS6 has similar improvement (taken in a simple from) over ÉGS5 as Glicko has over FIDE's Élo which accounts for a grade trust (or establishment) based on frequency of play (i.e., less trusted or established grades change more rapidly than more trusted or established grades, consequently, in extreme case, ungraded players do not affect the grades of graded players; uses logistic curve for 'p = f(d)'; does not stretch the grades).

Code: Select all

--------------------------------------------------------------
grading   stretches  uses FIDE   changes less     preserves 
system    grades     'p = f(d)'  trusted grades   total system
          ('k')      (yellow)    more rapidly     grade
--------------------------------------------------------------
GS        yes        no          no               yes
AGS3      no         no          no               yes 
ÉGS5      no         yes         no               yes
ÉGS6      no         yes         yes              no
--------------------------------------------------------------

Table 2: Main differences between GS (current Grading System), AGS3 (Amended Grading System three), ÉGS5 (Élo Grading System five) and ÉGS6 (Élo Grading System six).

Estimated effect on stretching...

Using a system where the sum of the factors 'k' is always equal to 1 (i.e. addressing the grade stretching problem) would affect the grades significantly in a longer run as in GS the grade stretching happens all the time (the effect increases with performance difference) and accumulates with time. Élo's logistic curve (present in both ÉGS5 and ÉGS6) wouldn't affect the grades significantly as its effect is relatively small for relatively small grade differences (the effect increases with grade difference), but it may affect the grades noticeably in cases where the grade difference is big. Professor Glickman's idea about grade establishment wouldn't affect the grades significantly in general as its effect is in general relatively small, but it may affect the grades significantly in cases where ungraded (or less active) players play graded (or more active) players.

[Peter Rhodes]

They are similar only in the sense that both are rating systems.

The similarity does not end there.

I think even the most casual observer will agree that the current system of "adding 50 for a win and subtracting 50 for a loss" is a strong similarity to the Harkness System.

To avoid pedantry, run this little script to any posts in this thread.
sed's/Harkness/Harkness derivative/g'

[Roger de Coverly]

I think even the most casual observer will agree that the current system of "adding 50 for a win and subtracting 50 for a loss" is a strong similarity to the Harkness System.

Well as the Harkness system used 4 figure numbers (1850 etc.) and the ECF system 3 figure numbers (150 etc.) , I would suggest to you that a Harkness 50 is equivalent to an ECF 5.

Basically both Harkness and Elo compute new grades as a weighted average of old grade and recent performance, whereas for a sufficiently high number of games, the ECF system computes new grades solely on recent performance.

[Paul McKeown]

wiki wrote

Amusing when wikipedia gets quoted as a reference work, rather than an original source.

As for the rest, Roger has comprehensively refuted it. Any similarities that might be drawn between a Harkness system and a Clarke system are skin deep at best.

[Peter Rhodes]

The purpose of quoting the Wikipedia is to draw the readers attention to the facts. Why would that be amusing.
The current system the ECF use which is CLEARLY a derivative of the Harkness system is used nowhere else in the world. That issurely a strong clue as to it's efficaciousness.

My issue is that the system of "adding 50 for a win and subtracting 50 for a loss" is a function that has no statistical basis and has to be fudged when the difference between the opponents is more than 40, making it not properly commutative.

It is so obvious that the flaws of the Harkness system are also present in the current ECF system. That is the similarity of which I speak. The accuracy of the system, or whether it uses 4 digits instead of 3 is just pure pedantry.

[Simon Spivack]

The purpose of quoting the Wikipedia is to draw the readers attention to the facts.

Resorting to Wikipedia for one's facts is amusing and a source of error, surely everyone knows this?

[Peter Rhodes]

Resorting to Wikipedia for one's facts is amusing and a source of error, surely everyone knows this?

When a player competes in a tournament, the average rating of his competition is calculated. If the player scores 50% he receives the average competition rating as his performance rating. If he scores more than 50% he new rating is the competition average plus 10 points for each percentage point above 50. If he scores less than 50% his new rating is the competition average minus 10 points for each percentage point below 50 (Harkness 1967:185-88).

Simon, you assert that this text from Wikipedia - describing the Harkness system - is innacurate ?