Some thoughts on anti-cheating systems

[JustinHorton]

The significance of this case was that statistical methods alone, far from determining that he was a cheat, could not even put a probability on this being true. That's because all statistical methods of which I am aware rely on differences between actual outcomes and expected outcomes. In this case, because the teenager had no previous 'form', there was no legitimate basis for calculating or even estimating expected outcomes. Without expected outcomes, there was no way of arriving at the difference between these and the actual outcomes. No data equates to no results.

I am not entirely happy that this is a correct conclusion, but I do not see how I challenge it without at least in some way discussing the case concerned.

Similarly, when people join Lichess, they are invited to supply a national or international grade/rating. Many newbies either won't have, or won't supply, this item of information so one has to ask - how does Lichess arrive at expected outcomes? Or, to put it another way, how do they know how well a totally unknown person plays chess?

This doesn't just apply to unknown players. One of my points during my struggle with chess.com was that they had no accurate idea of how well I played the form of chess in which I'd been taking part.

[Roger Lancaster]

The significance of this case was that statistical methods alone, far from determining that he was a cheat, could not even put a probability on this being true. That's because all statistical methods of which I am aware rely on differences between actual outcomes and expected outcomes. In this case, because the teenager had no previous 'form', there was no legitimate basis for calculating or even estimating expected outcomes. Without expected outcomes, there was no way of arriving at the difference between these and the actual outcomes. No data equates to no results.

I am not entirely happy that this is a correct conclusion, but I do not see how I challenge it without at least in some way discussing the case concerned.

Similarly, when people join Lichess, they are invited to supply a national or international grade/rating. Many newbies either won't have, or won't supply, this item of information so one has to ask - how does Lichess arrive at expected outcomes? Or, to put it another way, how do they know how well a totally unknown person plays chess?

This doesn't just apply to unknown players. One of my points during my struggle with chess.com was that they had no accurate idea of how well I played the form of chess in which I'd been taking part.

Justin, thanks for showing sensitivity on the first point. On the second, as I think you will already have worked out, I agree.

[John McKenna]

Roger L>The significance of this case was that statistical methods alone, far from determining that he was a cheat, could not even put a probability on this being true. That's because all statistical methods of which I am aware rely on differences between actual outcomes and expected outcomes.<

Don't know about that because -

The z-M test is a commonsense criterion expressed in probabilistic terms. In general, a performance (actual) may be regarded as exceptional if the probability of its is less than 10%, or better yet, less than 5%. (A. Elo)

E.g. an exception(al) performance over 9 games for a player with K=32 would be a gain (or loss) of 62 rating points at a 10% probability threshold and a gain (loss) of 79 rating points at a 5% probability threshold.

[Matthew Turner]

I'd be a tad more cautious than Matt and, since this is something I seem to have omitted from my earlier post in an effort to keep it to a manageable length, I'll try to explain why here. Some while ago, a previously completely unknown teenager emerged on the English chess scene playing to a standard where his ECF grade, converted to ELO, was a shade over 2400. Naturally, questions were asked. Further discussion on this forum was ruled prejudicial and the topic closed. I urge everyone not to reopen it as I cite it only as an extreme example of the relevant type of occurrence.

The significance of this case was that statistical methods alone, far from determining that he was a cheat, could not even put a probability on this being true. That's because all statistical methods of which I am aware rely on differences between actual outcomes and expected outcomes. In this case, because the teenager had no previous 'form', there was no legitimate basis for calculating or even estimating expected outcomes. Without expected outcomes, there was no way of arriving at the difference between these and the actual outcomes. No data equates to no results

You could have saved yourself some time and just written the last sentence. If you have the Pgns for a player’s games in an event you can perform a Regan test, if you don’t, you can’t.

[Roger Lancaster]

Roger L>The significance of this case was that statistical methods alone, far from determining that he was a cheat, could not even put a probability on this being true. That's because all statistical methods of which I am aware rely on differences between actual outcomes and expected outcomes.<

Don't know about that because -

The z-M test is a commonsense criterion expressed in probabilistic terms. In general, a performance (actual) may be regarded as exceptional if the probability of its is less than 10%, or better yet, less than 5%. (A. Elo)

E.g. an exception performance over 9 games for a player with K=32 would be a gain (or loss) of 62 rating points at a 10% probability threshold and a gain (loss) of 79 rating points at a 5% probability threshold.

John - put, perhaps more succinctly, one can't decide whether someone's performance is exceptional or not without knowing whether that person is a 7-year-old novice or Magnus Carlsen out to have fun under an anonymous username.

[Roger Lancaster]

I'd be a tad more cautious than Matt and, since this is something I seem to have omitted from my earlier post in an effort to keep it to a manageable length, I'll try to explain why here. Some while ago, a previously completely unknown teenager emerged on the English chess scene playing to a standard where his ECF grade, converted to ELO, was a shade over 2400. Naturally, questions were asked. Further discussion on this forum was ruled prejudicial and the topic closed. I urge everyone not to reopen it as I cite it only as an extreme example of the relevant type of occurrence.

The significance of this case was that statistical methods alone, far from determining that he was a cheat, could not even put a probability on this being true. That's because all statistical methods of which I am aware rely on differences between actual outcomes and expected outcomes. In this case, because the teenager had no previous 'form', there was no legitimate basis for calculating or even estimating expected outcomes. Without expected outcomes, there was no way of arriving at the difference between these and the actual outcomes. No data equates to no results

You could have saved yourself some time and just written the last sentence. If you have the Pgns for a player’s games in an event you can perform a Regan test, if you don’t, you can’t.

Maybe it's just the phraseology but I'm not sure we agree. If one has all a player's game scores in a particular event, but no more information about that player, the Regan test will tell you that he played to such-and-such a level but no more than that because it doesn't have the further information needed to determine the player's expected level and, consequently, what deviation there might be between the two.

[John McKenna]

4NCL OTB is a social as well as a competitive team league.

4NCL O/l is an asocial and therefore highly likely to be more competitive than the OTB league.

What?

Have you not heard of the infamous 4NCL OTB morning game after the social the night before??

4NCL O/l is a hunting ground for young wolf-warriors, by contrast.

[MartinCarpenter]

I assume I told Lichess my FIDE-rating, if they asked, but they still estimated me as 1500 when I started playing. I even got congratulated by a club colleague for beating an 1800...

There’s actually a non trivial danger here. There’s plenty of clubs who have gone online just now and are playing as closed groups with everyone starting at 1500.

The resulting grades are obviously entirely unreliable to misleading.

[Matthew Turner]

I'd be a tad more cautious than Matt and, since this is something I seem to have omitted from my earlier post in an effort to keep it to a manageable length, I'll try to explain why here. Some while ago, a previously completely unknown teenager emerged on the English chess scene playing to a standard where his ECF grade, converted to ELO, was a shade over 2400. Naturally, questions were asked. Further discussion on this forum was ruled prejudicial and the topic closed. I urge everyone not to reopen it as I cite it only as an extreme example of the relevant type of occurrence.

The significance of this case was that statistical methods alone, far from determining that he was a cheat, could not even put a probability on this being true. That's because all statistical methods of which I am aware rely on differences between actual outcomes and expected outcomes. In this case, because the teenager had no previous 'form', there was no legitimate basis for calculating or even estimating expected outcomes. Without expected outcomes, there was no way of arriving at the difference between these and the actual outcomes. No data equates to no results

You could have saved yourself some time and just written the last sentence. If you have the Pgns for a player’s games in an event you can perform a Regan test, if you don’t, you can’t.

Maybe it's just the phraseology but I'm not sure we agree. If one has all a player's game scores in a particular event, but no more information about that player, the Regan test will tell you that he played to such-and-such a level but no more than that because it doesn't have the further information needed to determine the player's expected level and, consequently, what deviation there might be between the two.

Well I just suggest you have a think about it. Work out if any pgns for a whole event exist in the case you are talking about. Pick a rating you want to use to start off and use your imagination to work out what the result will be.

[Roger de Coverly]

The significance of this case was that statistical methods alone, far from determining that he was a cheat, could not even put a probability on this being true. That's because all statistical methods of which I am aware rely on differences between actual outcomes and expected outcomes.

There was a result in one event which was so bad as to cast doubt on the legitimacy of all the others.

For all we know lichess and chess.com ban players for attempted sandbagging.

[Matthew Turner]

Roger,
What like actually looking at their terms and conditions?

From Lichess

See the following for some examples of what you shouldn’t do.

2. Artificially inflating or deflating your rating. This is where a User purposefully loses, or has arranged with an opponent to win. As a result, the User’s rating will artificially increase or decrease.

[Roger Lancaster]

Well I just suggest you have a think about it. Work out if any pgns for a whole event exist in the case you are talking about. Pick a rating you want to use to start off and use your imagination to work out what the result will be.

Over-the-board events, if one can be reasonably sure that a player isn't cheating, are one thing - one can then assess 'true playing strength' from someone's moves. However, I was talking about moves played in an online tournament where - for all Lichess et al know - someone could, for example, have had a grandmaster looking over their shoulder and guiding them. Those games wouldn't form a sound basis for expected playing strength.

[Matthew Turner]

Roger,
If I understand you correctly, you are now talking about a hypothetical example which has nothing to do with anything that is happening in British Chess or anything that has happened historically. You’ve got me, I don’t know.

[John McKenna]

I'd like to repeat - if we all could check our online anti-cheating credit/debit scores with the various chess orgs it might be a place to start!?

It may also be worth repeating (not sure about everyone seeing everyone else's details but in at least one Scandinavian country that is possible regarding pay, I believe) -

The session last night was very interesting.

I don't understand why GDPR prevents the 4NCL publishing the Z-scores of all the players in the 4NCL online. Any publication should come with a disclaimer that the Z-scores are statistics only and whilst they may create a very strong statistical inference that computer assistance was used, they don't prove that it was.

The players names, ratings, and the game scores are already in the public domain. As I understand it, these are the main inputs that Regan uses to calculate the Z-scores, and then he just some fancy analysis of them, using a method that may or may not be in the public domain as well. There is a legitimate interest in processing this data to detect cheating, and presumably it could be replicated by anyone understood Regan's method sufficiently well to replicate it.

Alternatively, if publishing Z-scores breaches GDPR, then why doesn't publishing ECF Grades, FIDE ratings, Chess.com Move Accuracy Scores, or LiChess centipawn loss scores, for any players who are named by their real name? As they are all essentially the same thing i.e. numbers calculated on the basis of raw game data.

[Richard Bates]

I'd like to repeat - if we all could check our online anti-cheating credit/debit scores with the various chess orgs it might be a place to start!?

It may also be worth repeating (not sure about everyone seeing everyone else's details but in at least one Scandinavian country that is possible regarding pay, I believe) -

We could ditch any need for a grading system as well! "Z-score" would be the new test of playing strength

OK (and note I haven't paid anything but the most cursory attention to what it actually is) so there is the problem potentially of people playing deliberately bad moves in completely winning positions, without actually jeopardising the chances of a positive result...