GRADING ANOMALIES

General discussions about ratings.
Brian Valentine
Posts: 574
Joined: Fri Apr 03, 2009 1:30 pm

Re: GRADING ANOMALIES

Post by Brian Valentine » Thu Dec 17, 2009 12:43 pm

I have posted a note on bias is the other thread - another way of looking at the ecf grading method.

Roger de Coverly
Posts: 21301
Joined: Tue Apr 15, 2008 2:51 pm

Re: GRADING ANOMALIES

Post by Roger de Coverly » Thu Dec 17, 2009 1:30 pm

Robert Jurjevic wrote:Could somebody please summarize the two component idea of "estimate of strength" plus a "bias or error"? Thanks.
Chess players have known for many years that all grades are wrong. It's not hard to see why as the measurement system ignores the 29 Carlsen-standard moves and concentrates its measurement on the 30th when you left your queen en prise during time trouble. So the grading system is reflecting both chess strength and the ability to clock manage.

If a grade is a measurement of strength using results as a proxy, then there's likely to be a built in error. If the intention of a grading system is to rank players in order of strength, then one of its features has to be error minimisation and correction.

User avatar
Robert Jurjevic
Posts: 207
Joined: Wed May 16, 2007 1:31 pm
Location: Surrey

Re: GRADING ANOMALIES

Post by Robert Jurjevic » Thu Dec 17, 2009 2:06 pm

Hello Brian,
Brian Valentine wrote:I have posted a note on bias is the other thread - another way of looking at the ecf grading method.
Thanks. (the post in question is http://www.ecforum.org.uk/viewtopic.php?f=4&t=1038)

Elements of your matrix 'S' for 'ka + kb = 2' grading systems are (approximately) double of that for 'ka + kb = 1' grading systems. I suspect (but not sure) that the reason for 'grade stretching' could be the fact that (as currently is) 'ka + kb = 2' and that it may be that it should be 'ka + kb = 1', i.e. that it may be that the elements of matrix 'S' should be (approximately) halved.

I wouldn't give so much attention to 'p^*' which you refer to as a quantity which can be interpreted from a description of the process used by the graders, IMHO p^*' is basically (with few exceptions for 'special' player categories) 'p^0' which you refereed to as to the available prior year grade. IMHO a better approach would be to deal with elements of matrix (or matrices) 'S', as they should IMHO contain the rules how to correct grades of all players, either un-graded players, fast improving juniors, etc., all this can be handled by the choice of 'k' factors for each game and 'p = f(d)' relation (which both appear in the elements of your matrix 'S'), also I do not think that the player categories should be chosen so coarse, say I do not care if somebody is a junior or not, what I care about is how fast he is improving (or worsening), also why only distinguish between un-graded and graded players, when grades of all graded players may not be equally trusted (my proposal in ÉGS6 where you have gradation of grade trust based on frequency of play with un-graded players being treated as the players whose grades are the least trusted), etc. (elements of your matrix 'S' are in principle functions of 'k' factors, 'p = f(d)' relation and 'q')

I would assume that it would be difficult to better my proposal for grading un-graded players given in "Ungraded players..." section at the end of http://www.jurjevic.org.uk/chess/grade/ ... malies.htm document.

Just an irrelevant observation... can't an element of matrix 'N' be say 2/m(i), if 'i'th player played 2 games against 'j'th player (say sometimes I play 2 games against the same opponent in a season, say Surrey Border League matches often have the same players on the same boards in home and away matches)?

Thanks.

Kind regards,
Robert Jurjevic
Vafra

Brian Valentine
Posts: 574
Joined: Fri Apr 03, 2009 1:30 pm

Re: GRADING ANOMALIES

Post by Brian Valentine » Thu Dec 17, 2009 3:23 pm

Robert,
We are at cross purposes. I am trying to test the ecf method, not find an alternative. I am setting out how it currently works. If the ecf moves then ELO is the natural direction.

Thank you for pointing out what was my misuse of language. you are right about 2/m. "Invariably" should be replaced by "almost always"

User avatar
Robert Jurjevic
Posts: 207
Joined: Wed May 16, 2007 1:31 pm
Location: Surrey

Re: GRADING ANOMALIES

Post by Robert Jurjevic » Thu Dec 17, 2009 4:07 pm

Hello Brian,
Brian Valentine wrote:We are at cross purposes. I am trying to test the ecf method, not find an alternative. I am setting out how it currently works. If the ecf moves then ELO is the natural direction.
Right.

I just wondered (if you would be interested) would you be able to tell (using your 'matrix method' and a mathematical analysis) what would happen to the grading system if the elements of your matrix 'S' are halved (juniors, un-graded players and any other 'special' players should be ignored)?

A measure of how much the grades calculated using two grading systems are 'stretched' in respect to each other may be related to the standard deviations of the grade distributions (say one may expect that grades with larger standard deviation are 'stretched' in respect to grades with smaller standard deviation).

I expect that the standard deviation of the grade distribution for the grading system with current matrix 'S' would be larger than the standard deviation of the grade distribution for the grading system with matrix 'S' with halved elements.

My numerical analysis for 2009 shows (if standard deviation is a measure of 'grade stretching') that GS grades should be 'stretched' in respect to AGS3 grades (GS standard deviation is 37.05 and AGS3 standard deviation is 36.73).

Image
Figure 1: Histogram for 2009 GS grades (consisting of 10153 grades in bins [0,1), [1,2), etc., of 1 grading point width, where minimum grade is 0.00, median grade is 133.00, maximum grade is 281.00, mean grade is 133.69 and standard deviation is 37.05).

Image
Figure 2: Histogram for 2009 AGS3 grades (consisting of 10153 grades in bins [0,1), [1,2), etc., of 1 grading point width, where minimum grade is 0.00, median grade is 133.50, maximum grade is 281.00, mean grade is 133.55 and standard deviation is 36.73).

Now I see that comparing standard deviations of the grade distributions is probably not a good method of assessing if one grading system 'stretches' grades in respect to the other, the reason is that (at least in theory) one can have two different grading systems each producing histograms with the same number of bins and the same number of players in each bin (then the standard deviations of the two grade distributions would be equal) and the only thing in what the systems may differ is which players are put in the bins.

Thanks.

Kind regards,
Last edited by Robert Jurjevic on Thu Dec 17, 2009 6:08 pm, edited 4 times in total.
Robert Jurjevic
Vafra

Brian Valentine
Posts: 574
Joined: Fri Apr 03, 2009 1:30 pm

Re: GRADING ANOMALIES

Post by Brian Valentine » Thu Dec 17, 2009 4:52 pm

Robert,

They are not my matrices, they are an outcome of the grader's method.

If one ignores the junior complication the both the N and S matrices simplify to the RR portions

Pr = Nrr p0 + Srr 1 from equation 3 (hope the change from superscripts makes sense - now I'll drop them altogether)

And in equation 4 (rr bit)

E [S.1] = I g – N g

Halving all items in S would mean you have half the expected value - I'm struggling to see how this makes any useful contribution. The ECF system meets the first requirement of a system in that


E(P1) = NP0+ (I - N)g

= g + N (P0-g)

If there no no bias initial (p=g) then the expected grade is equal to underlying strength. If S is changed then g has to be halved to meet this basic statistical requirement. You will note I used 1% of the proper numbers throughout halving everything doesn't do anything useful.

Given that the estimates of uu are also unbiased, then the new ECF calibration gets over this first hurdle in that it is not mechanistically introducing bias. However there are many more tests to consider.

User avatar
Robert Jurjevic
Posts: 207
Joined: Wed May 16, 2007 1:31 pm
Location: Surrey

Re: GRADING ANOMALIES

Post by Robert Jurjevic » Thu Dec 17, 2009 9:33 pm

Hello Brian,
Brian Valentine wrote:Halving all items in S would mean you have half the expected value - I'm struggling to see how this makes any useful contribution.
Let us assume that there are 'n' players who play each other only once (every player plays all players exactly once). If 'g' is a grade vector where 'gi' is grade of 'i'th player, 'Q' 'nxn' matrix where 'qij' is actual performance of 'i'th player in the game against 'j'th player (0 if 'i'th player lost, 50 if 'i'th player drew and 100 if 'i'th player won, note that 'qij' and 'qji' are related, i.e., if 'qij=0' then 'qji=100','if 'qij=50' then 'qji=50', if 'qij=100' then 'qji=0', note that 'qij+qji=100'), 'P' 'nxn' matrix where 'pij = f(gi-gj)' is expected performance of 'i'th player in the game against 'j'th player (say if 'gi=gj' then 'f(gi-gj)=50', note that 'pij+pji=100', please note that relationship 'pij=f(gi-gj)') need not to be linear, it can be logistic, then new grades vector 'g2' is (sums go for every 'i' and 'j' such that 'i\=j')

g2i = gi + Sum_j ((kij*(qij - pij))/n) = gi + Sum_j ((kij*qij)/n) - Sum_j ((kij*pij)/n) = gi + Sum_j (sij*gj)

where 'K' is 'nxn' matrix such that 'kij+kji=k', where 'k' is a constant, and 'S' your matrix which can be solved from

'Sum_j ((kij*qij)/n) - Sum_j ((kij*pij)/n) = Sum_j (sij*gj)'.

I think that it is possible that the constant 'k' in 'kij+kji=k' should be 1 rather than 2 (which is the case now).

Say that you look at the game between players 2 and 3, then the correction applied to player 2 is '(k23*(q23 - p23))/n' and the correction applied to player 3 is 'k32*(q32 - p32))/n', as '(qij - pij) + (qji - pji)' so is '(q23 - p23) + (q32 - p32) = 0' and '|(q23 - p23)| = |- (q32 - p32)| = c', where 'c' is the absolute value of the difference between actual and expected performance for players 2 and 3 for the game, assuming that 'k23 > 0', 'k32 > 0' and 'n > 0' the total absolute correction for the game is 'k23*c/n + k32*c/n = (k23 + k32)*c/n', as 'c' is the absolute value of the difference between actual and expected performance for players 2 and 3 for the game it looks to me that it should be 'k23 + k32 = k = 1' rather than 'k23 + k32 = k = 2', as then the total absolute correction for the game would be 'c/n' rather than '2*c/n', i.e. like 'c/n' is total available correction one should apply to the players for the game, which means if you correct grade of player 2 for the game for 'c/n' then you should not correct the grade of player 3 and vice versa, AGS3 would correct the grades of both players for the game for 'c/2/n', GS would correct the grades of both players for the game for 'c/n', in my opinion, possibly over-correcting the grades of the two players for 'c/n'.

If you think that the 'k' constant should be 2 rather than 1 could you please explain why? Thanks.

(please note that 'K' matrix contains 'k' factors applied in each game for each player, 'kij' is 'k' factor applied to grade correction of 'i'th player for game against 'j'th player, 'Q' matrix represents players' actual performances, 'qij' is actual performance of 'i'th player in the game against 'j'th player, 'P' matrix represents players' expected performances, 'pij' is expected performance of 'i'th player in the game against 'j'th player, matrix 'S' can be regarded as a grade correction operator, the larger 'k' in 'kij+kji=k' the larger 'S' elements, as 'Sum_j ((kij*(qij - pij))/n) = Sum_j (sij*gj)', and the larger the grade correction; my point is that the current grading system may apply too much of a grade correction, i.e., that it is possible that 'S' matrix elements are chosen to be too big, in fact instead of 'k=2' in 'kij+kji=k' one may need to chose 'k=1' which effectively means that one may need to halve 'sij' elements)

Kind regards,
Robert Jurjevic
Vafra

Roger de Coverly
Posts: 21301
Joined: Tue Apr 15, 2008 2:51 pm

Re: GRADING ANOMALIES

Post by Roger de Coverly » Thu Dec 17, 2009 10:06 pm

Robert Jurjevic wrote:Let us assume that there are 'n' players who play each other only once (every player plays all players exactly once).
Not a real world assumption. In practice you are playing 30 players out of a universe of 10000. The grading matrix is sparsely populated really. As far as I am concerned, the main objection to using +/-25 instead of +/-50 is the pragmatic one that it gives the wrong answers for player rankings where a start season grade has a large bias/error/lag factor in it.

Brian Valentine
Posts: 574
Joined: Fri Apr 03, 2009 1:30 pm

Re: GRADING ANOMALIES

Post by Brian Valentine » Fri Dec 18, 2009 9:50 am

Robert,
There is no reason why you should adopt my definitions, but using the same letters for different matrices makes it difficult to comment. Let's see if understand your point, ignoring the logistic linear issue and my currency being 1% of yours, for clarity.

I think your Q matrix is my W matrix (apart from mine potentially handling matches).

I'm not sure if your g vector is my p0 or my g or even E(p) - I read it differently in different places

Your P matrix is D-M in the ecf world?

I cannot see where the k enters into the ecf formulae. The elo k exists in its system and as RdC has explained the concept exists in the ECF world, but only to explain how it compares with ELO.

If you could help me on these definitions I'll have another go at understanding your post.

Roger de Coverly
Posts: 21301
Joined: Tue Apr 15, 2008 2:51 pm

Re: GRADING ANOMALIES

Post by Roger de Coverly » Fri Dec 18, 2009 10:50 am

Brian Valentine wrote:If you could help me on these definitions I'll have another go at understanding your post.
Robert's fundamental belief is that it's wrong that when a 120 player draws with a 160 player that the 120 player gets a contribution of 160/n to his next grade whereas the 160 player gets a contribution of 120/n. He believes that both should get 140/n despite the inconsistency with measurements against "new" players ,the added lag this creates for players with "bias" and the issue that a measurement of performance should be independent of previous performance. It's as if an athlete runs a 100 metres in 10 seconds but it only counts as 10.5 seconds because his previous best was 11 seconds.

Ian Thompson
Posts: 3551
Joined: Wed Jul 02, 2008 4:31 pm
Location: Awbridge, Hampshire

Re: GRADING ANOMALIES

Post by Ian Thompson » Fri Dec 18, 2009 11:18 am

Roger de Coverly wrote:Robert's fundamental belief is that it's wrong that when a 120 player draws with a 160 player that the 120 player gets a contribution of 160/n to his next grade whereas the 160 player gets a contribution of 120/n. He believes that both should get 140/n despite the inconsistency with measurements against "new" players ,the added lag this creates for players with "bias" and the issue that a measurement of performance should be independent of previous performance. It's as if an athlete runs a 100 metres in 10 seconds but it only counts as 10.5 seconds because his previous best was 11 seconds.
Measurements of chess players' and athletes' performances can't be compared. The time an athlete runs the 100m in is an absolute measure of his performance, so its easy to rank the performances of different athletes. When a 120 graded chess player draws with a 160 graded chess player you have no idea whether the 120 player played to 160 standard in the game, the 160 player played to 120 standard in the game, or they both played to any other standard you like from complete beginner to grandmaster.

User avatar
Robert Jurjevic
Posts: 207
Joined: Wed May 16, 2007 1:31 pm
Location: Surrey

Re: GRADING ANOMALIES

Post by Robert Jurjevic » Fri Dec 18, 2009 11:58 am

Hello all,
Roger de Coverly wrote:
Robert Jurjevic wrote:Let us assume that there are 'n' players who play each other only once (every player plays all players exactly once).
Not a real world assumption. In practice you are playing 30 players out of a universe of 10000.
Okay, fair enough. I will make a model which will have no assumptions or restrictions regarding who can play who (I will also allow matches between two players though they need not to happen).

Restrictions:

o I will assume that we are grading only players who played at least 30 games in the season (later I may lift this restriction, but for the moment I would wish to have it so that the main idea is expressed as clear as possible)

Definitions:

o 'n' is the total number of players in the system

o 'n_' is game number vector where 'ni' is a number of games 'i'th player played in the season

o 'g_' is a grade vector where 'gi' is grade from previous season of 'i'th player

o 'g2_' is a new grade vector where 'g2i' is a new grade for the season of 'i'th player (player's new grade for the season is calculated using the player's game results in the season and the player's and the player's opposition grades from previous season)

o 'Q_' 'is nxn' matrix where 'qij' is actual performance of 'i'th player in the game against 'j'th player (0 if 'i'th player lost, 50 if 'i'th player drew and 100 if 'i'th player won, note that 'qij' and 'qji' are related, i.e., if 'qij=0' then 'qji=100','if 'qij=50' then 'qji=50', if 'qij=100' then 'qji=0'), note that 'qij+qji=100' if 'i'th and 'j'th players played at least one game in the season, note that 'qij = qji = 0' if 'i=j' or if 'i'th and 'j'th player did not play each other in the season, if the players 'i' and 'j' played more than one game than '0<=qji<=100', 'Q_' is actual performance matrix

o 'P_' is 'nxn' matrix where 'pij=f(gi-gj)' is expected performance of 'i'th player in the game against 'j'th player, note that ECF is using 'f(dij=gi-gj)=50*(1 + (gi-gj)/50)' if '|dij=gi-gj|<=40' and 'f(dij=gi-gj)=90' if 'dij=gi-gj>40' and 'f(dij=gi-gj)=10' if 'dij=gi-gj<-40', FIDE logistic equivalent is 'f(dij=gi-gj)=100/(1 + 10^(-(gi-gj)/50))', note that if 'gi=gj' then 'f(gi-gj)=50', note that 'pij+pji=100', 'P_' is expected performance matrix

o 'K_' is 'nxn' matrix where 'kij' are 'k' factors used in calculation of new grades for the season, 'kij=kji=0' if 'i=j' or if 'i'th and 'j'th player did not play each other in the season, 'kij+kji=k', where 'k>0' is a constant, if 'i'th and 'j'th players played at least one game in the season, 'K_' is 'k' factors matrix

o 'S_' is 'nxn' matrix where 'sij' are terms used in calculation of new grades for the season, 'S_' is Brian's matrix

New grades in the season are calculated as follows (this is a generic formula covering a number of grading systems including the current ECF one, what makes the grading systems different is a choice of matrices 'K_' and 'P_')

Code: Select all

g2i = gi + Sum_j ((kij*(qij - pij))/ni) 
where for current ECF grading system 'kij=kji=1' if 'i'th and 'j'th players played at least one game in the season and 'pij=f(gi-gj)' where 'f(dij=gi-gj)=50*(1 + (gi-gj)/50)' if '|dij=gi-gj|<=40' and 'f(dij=gi-gj)=90' if 'dij=gi-gj>40' and 'f(dij=gi-gj)=10' if 'dij=gi-gj<-40'.

First obvious improvement of the current ECF grading system would be to use FIDE logistic equivalent for 'f(gi-gj)', i.e. to use 'pij=f(gi-gj)' where 'f(dij=gi-gj)=100/(1 + 10^(-(gi-gj)/50))'.

After long thinking and experimenting (and with a lot of help form people of this forum) I came to a conclusion that what might be causing 'grade stretching' is the fact that the constant 'k>0' in matrix 'K' is 2 rather than 1, what I think it may be possible that it should be (for current ECF grading system 'kij=kji=1' if 'i'th and 'j'th players played at least one game in the season, 'kij=kji=0' if 'i=j' or if 'i'th and 'j'th player did not play each other in the season).

Brian's formula for calculating new grades in the season I understand is

Code: Select all

g2i = gi + Sum_j (sij*gj) 
where 'sij' are elements of Brian's matrix 'S_' which can be solved from

Code: Select all

Sum_j ((kij*(qij - pij))/ni) = Sum_j (sij*gj) 
Brian Valentine wrote:I think your Q matrix is my W matrix (apart from mine potentially handling matches).
Sorry, I couldn't find a mention of 'W' matrix in your article ecfratinginflation.pdf at http://www.ecforum.org.uk/viewtopic.php?f=4&t=1038. 'Q_' matrix is actual performance matrix.
Brian Valentine wrote:I'm not sure if your g vector is my p0 or my g or even E(p) - I read it differently in different places.
Sorry, I couldn't find a mention of 'g' vector nor 'E(p)' in your article ecfratinginflation.pdf at http://www.ecforum.org.uk/viewtopic.php?f=4&t=1038. Yes, my 'g_' vector should be your 'p0' vector.
Brian Valentine wrote:Your P matrix is D-M in the ecf world?
Sorry, I couldn't find a mention of 'D' nor 'M' matrix in your article ecfratinginflation.pdf at http://www.ecforum.org.uk/viewtopic.php?f=4&t=1038. 'P_' matrix is expected performance matrix.
Brian Valentine wrote:I cannot see where the k enters into the ecf formulae. The elo k exists in its system and as RdC has explained the concept exists in the ECF world, but only to explain how it compares with ELO.
'g2i = gi + Sum_j ((kij*(qij - pij))/ni)' is a generic formula which covers all grading systems I am examining, the formula is the current ECF formula if 'kij=kji=1' if 'i'th and 'j'th players played at least one game in the season and if 'pij=f(gi-gj)' where 'f(dij=gi-gj)=50*(1 + (gi-gj)/50)' if '|dij=gi-gj|<=40' and 'f(dij=gi-gj)=90' if 'dij=gi-gj > 40' and 'f(dij=gi-gj)=10' if 'dij=gi-gj < -40', so as you see the current ECF calculation method also uses 'K_' matrix.

If you would agree I would suggest that we continue discussing why one may wish to set constant 'k>0' in matrix 'K_' to 1 rather than to 2 once we accept the above model and become familiar with nomenclature and terminology used.

Kind regards,
Last edited by Robert Jurjevic on Fri Dec 18, 2009 2:15 pm, edited 5 times in total.
Robert Jurjevic
Vafra

Brian Valentine
Posts: 574
Joined: Fri Apr 03, 2009 1:30 pm

Re: GRADING ANOMALIES

Post by Brian Valentine » Fri Dec 18, 2009 12:20 pm

Sorry Robert, the document I was referring to is the one I added on 17th December: ecf ratingon biasinfluence.pdf [url][http://www.ecforum.org.uk/viewtopic.php ... p21121/url]. I'll work with your definitions for know though.

User avatar
Robert Jurjevic
Posts: 207
Joined: Wed May 16, 2007 1:31 pm
Location: Surrey

Re: GRADING ANOMALIES

Post by Robert Jurjevic » Fri Dec 18, 2009 12:55 pm

Roger de Coverly wrote:Robert's fundamental belief is that it's wrong that when a 120 player draws with a 160 player that the 120 player gets a contribution of 160/n to his next grade whereas the 160 player gets a contribution of 120/n. He believes that both should get 140/n despite the inconsistency with measurements against "new" players, the added lag this creates for players with "bias" and the issue that a measurement of performance should be independent of previous performance.
My fundamental belief is that it's wrong that when a 120 player draws with a 160 player that the 120 player is rewarded for 40/n grading points and the 160 player penalized for 40/n grading points. I believe that in genral the 120 player should be rewarded for 20/n grading points and the 160 player penalized for 20/n grading points, but my belief also allows for the case where the 120 player would be rewarded for 40/n grading points and the 160 player neither penalized nor rewarded.
Roger de Coverly wrote:...and the issue that a measurement of performance should be independent of previous performance.
From my previous post... New grades in the season are calculated as follows (it is a generic formula covering a number of grading systems including the current ECF one, that what differs the grading systems is the choice of matrices 'K_' and 'P_')

Code: Select all

g2i = gi + Sum_j ((kij*(qij - pij))/ni) 
where for current ECF grading system 'kij=kji=1' if 'i'th and 'j'th players played at least one game in the season and 'pij=f(gi-gj)' where 'f(dij=gi-gj)=50*(1 + (gi-gj)/50)' if '|dij=gi-gj|>40' and 'f(dij=gi-gj)=90' if 'dij=gi-gj > 40' and 'f(dij=gi-gj)=10' if 'dij=gi-gj < -40'.

So, ECF currently does correct 'gi' for 'Sum_j ((kij*(qij - pij))/ni) ', i.e., one's new grade 'g2i' depends on one's old grade 'gi'.

I have already mentioned a few times (though I might have not been clear) that the current ECF choice of matrix 'P_' in the above formula (I again refer to my previous post) is such that 'g2i' becomes independent of 'gi' (as 'gi' when combined with the current ECF 'pij' becomes 'gj' where 'j/=i'), but say if one would choose a FIDE equivalent for 'pij' then 'g2i' would be a function of 'gi', so at least according to the general formula a new grade should be dependent on the old grade. FIDE corrects ratings, so I assume that a new FIDE rating should be dependent on the old FIDE rating.
Roger de Coverly wrote:It's as if an athlete runs a 100 metres in 10 seconds but it only counts as 10.5 seconds because his previous best was 11 seconds.
I would say it is more as the athlete runs a 100 metres in 10 seconds but as he ran in previous runs slower his average time of say 10.5 seconds is a measure if he will get into Olympic team or not, as you would not wish to judge the athlete's performance based on one run only. Makes some sense?
Last edited by Robert Jurjevic on Fri Dec 18, 2009 1:58 pm, edited 14 times in total.
Robert Jurjevic
Vafra

User avatar
Robert Jurjevic
Posts: 207
Joined: Wed May 16, 2007 1:31 pm
Location: Surrey

Re: GRADING ANOMALIES

Post by Robert Jurjevic » Fri Dec 18, 2009 1:07 pm

Brian Valentine wrote:Sorry Robert, the document I was referring to is the one I added on 17th December: ecf ratingon biasinfluence.pdf [url][http://www.ecforum.org.uk/viewtopic.php ... p21121/url]. I'll work with your definitions for know though.
Okay, thanks, could you please let me know when you and Roger are ready (I won't be able to reply on Saturday, 18th of December)? Thanks a lot.
Robert Jurjevic
Vafra