Some thoughts on anti-cheating systems

[Mike Gunn]

If you don’t want to wade your way through all of this I’m basically backing up OP Roger’s points that statistical inference may not work so well when applied to events resulting from human decisions.

I wonder if anyone here has read any of the works of Nassim Nicholas Taleb (Professor of Risk Engineering at New York University). I have read “The Black Swan” and got part of the way through “Fooled by Randomness” before deciding it was largely repetitive. His basic point is that you can’t apply the normal distribution of statistical theory to events where human decisions play a significant part. His work has been mainly influential in the finance industry where the calculation of risks of particular investments failing which (traditionally) used the techniques we have been discussing here.

I am interested in this field from having to teach modern codes of design to civil engineers. In this context an engineering designer calculates the statistical variation of resisting forces (e.g. supporting a bridge structure) depending on the statistical variation of the strengths of the bridge materials (steel, concrete etc.). He then calculates the statistical variation of the loads (e.g. traffic and wind loading) which will tend to cause the bridge to collapse. Comparing the resistances and the loads he can calculate the probability of the bridge collapsing in a life of (say) 100 years – a probability of 0.001 might be considered acceptable. The problem I have with this approach is that we have some data on bridge collapse and the majority of collapses take place not because the material strengths and loads reach their statistical limits but because the designers have designed some features very poorly (e.g. the joints between members) or haven’t anticipated some mode of bridge behaviour not included in the design (e.g. torsional oscillations as in the Tacoma Narrows bridge, imperfection sensitive buckling of flat plates in the Yarra and Milford Haven bridges. I should add that I am one of three professional engineers I know that believe the application of these statistical techniques is rubbish so I’m in a very small minority in the engineering profession.

Now I must admit that although I have downloaded Ken Regan’s papers I haven’t (yet) got round to reading them, so I’m not really in the area of being able to do a proper critique of his method, but when I heard him going on about Z values in Tuesday’s webinar I realised that it is all based on the area under the tail of a normal (or similar statistical) distribution. These calculations are all very well when it comes to truly random physical events (coin tossing etc) but perhaps not to events based on human decision taking.

[Roger de Coverly]

(edit) There's this stuff from around six years ago.
viewtopic.php?f=31&t=5085&p=138476#p138476 (/edit)

Reading some of that stuff again, it's the same arguments or similar arguments about the reliability of statistical tests as in this thread.

FIDE were persuaded, possibly by default, not to take action exclusively on the results of statistical tests. What we got in practice was the more sensible measure that mobile phones and similar devices should be completely out of use during play and seen to be so. The problem with online play is that is considerably more difficult to demonstrate. For that matter so is demonstrating the absence of the pre-computer methods of taking advice from a stronger spectator or consulting a book for assistance.

[Nick Ivell]

Let me attempt to summarise in four simple words:

ONLINE CHESS IS SKITTLES.

Excuse the upper case.

[Roger Lancaster]

Roger,
If I understand you correctly, you are now talking about a hypothetical example which has nothing to do with anything that is happening in British Chess or anything that has happened historically. You’ve got me, I don’t know.

Matt - You evidently don't. I'm not discussing a hypothetical example but actual examples which occur hundreds, possibly thousands, of times each day when someone signs up to Lichess. Since Lichess frequently have no idea who that person is, they can have no idea of his or her playing strength. Without this, they can make no remotely accurate assumption of the 'expected value' in the statistics.

[Roger de Coverly]

Since Lichess frequently have no idea who that person is, they can have no idea of his or her playing strength. Without this, they can make no remotely accurate assumption of the 'expected value' in the statistics.

If I understand it correctly, the 4NCL have the use of a black box which when run against a file of games will make allegations of engine assistance. Also when run against a traditional weekend, it didn't make any. However when run against the on-line version, containing an overlap of players, it did.

What I would ask is that if you told it every player in the traditional format had a rating of 1000, how many allegations would it make?

[Roger Lancaster]

Since Lichess frequently have no idea who that person is, they can have no idea of his or her playing strength. Without this, they can make no remotely accurate assumption of the 'expected value' in the statistics.

If I understand it correctly, the 4NCL have the use of a black box which when run against a file of games will make allegations of engine assistance. Also when run against a traditional weekend, it didn't make any. However when run against the on-line version, containing an overlap of players, it did.

What I would ask is that if you told it every player in the traditional format had a rating of 1000, how many allegations would it make?

Agreed. At heart, I believe that's the same point that I've been making.

[John McKenna]

Let me attempt to summarise in four simple words:

ONLINE CHESS IS SKITTLES.

Excuse the upper case.

Well, most "online chess" may have been "skittles" up to about when over-the-board chess began to stop from mid-March on.

Once otb chess ceased altogether it started to become more and there must have been a significant infux of players into playing online who had previously only played otb chess.

This and other related threads seem to be about trying to establish how chess played online can be brought up to a standard that the newly recruited otb players will accept as being worth taking seriously enough to play with a reasonable degree of confidence that they will only encounter cheating relatively rarely when playing online in 4NCL & ECF events.

One thing seems clear - if players had to identify themselves unequivocally by playing under usernames that matched their real names it would help dispell a large cloud of doubt.

Another big cloud of doubt is the one cast by the use of "algorithms" to determine if a player is using any assistance to play games online. With the additional problem that innocent players can be punished along with the guilty because the algorithms are not foolproof.

There are no easy answers or ways and means to bring online chess up to otb standards. Whatever might be gained from the ease and convenience of playing online (particularly in the current crisis) are offset by numerous suspicions and a general lack of trust that need to be allayed somehow.

If that can be done to some meaningful degree then players might accept that the remaining problems and pitfalls of playing online are worth the effort.

If not we will just have to await the return of otb chess to have a right and proper game (even if it kills us).

PS Thanks to RdC for digging up that old thread in which Ken Regan kindly, and with great patience and aplomb, explains himself. (Must read more.)

[Matthew Turner]

Roger,
If I understand you correctly, you are now talking about a hypothetical example which has nothing to do with anything that is happening in British Chess or anything that has happened historically. You’ve got me, I don’t know.

Matt - You evidently don't. I'm not discussing a hypothetical example but actual examples which occur hundreds, possibly thousands, of times each day when someone signs up to Lichess. Since Lichess frequently have no idea who that person is, they can have no idea of his or her playing strength. Without this, they can make no remotely accurate assumption of the 'expected value' in the statistics.

Roger, this has nothing to do with the 4NCL or the ECF where we have established grades for the players. Is what you describing a problem for Lichess, well in a word Yes. There are two points that I would make
1. Titled players are caught cheating, so your cheat would have to be adept enough to avoid using assistance to the extent that they would be flagged up regardless of their rating.
2. Major events are limited to titled players so everyone has an established OTB history

Now you have moved on to thinking about how Lichess can improve their systems for other events, I take it you are completely happy with the ECF fair play policy.

[MartinCarpenter]

Something immediate that Ken said in his first post to that old thread (my italics):
"The 20-year estimate for a 1M-to-1 false positive (using 4.75 sigma as criterion) is based on about 200,000 tested games per year, not 50,000, because the unit of testing is player-performance-in-one-event, not individual games. One can figure 8 games per event on average,"

This sort of thing matters.

If you're prepared to convict people based on just a small handful of games, then you're exposed to all sorts of potential problems to do with false positives.

If you note the suspicion, wait a bit and collect a decent sample your statistical confidence in the conviction becomes enormously stronger.

And what do you lose? Fundamentally there is no particular hurry to catch most cheats. They'll keep cheating. If they don't then problem solved.

For specific prestigious/prize events you obviously need to protect those events.

[Mick Norris]

If you want to play an event like the ECF Online Counties Championship, then when you have a knockout stage, you need to ctach any cheating before the next round, if it affects the overall result this is crucial

[Matthew Turner]

Mick,
You cannot do that in OTB events so why do you expect to be able to do it in online events?

[Mick Norris]

Good question; I suppose OTB that it is pretty clear at the end of a county match what the result is i.e. you are unlikely to discover cheating after the event, although you might get questions about eligibility of players

As you're saying that cheating online is much more frequent, it would seem an obvious concern there: I don't play much these days, but I haven't had any suspicion OTB at all of my opponents

[Matthew Turner]

Mick,
You have identified a problem and I don’t know the answer. Maybe in years to come county matches will be a different format, I.e each player playing 6 games, so that it is likely that a player will build up enough evidence after one round to detect cheating.

[Ian Thompson]

You may well be right. I sometimes fail to differentiate correctly between ECF and 4NCL [my apologies to anyone offended] so I should perhaps clarify that my understanding - and I can't be absolutely sure whether I asked the question directly at the time - was that it was a 4NCL ban rather than an ECF ban.

Ultimately, I think the differentiation will become meaningless because all graded events will become ‘ECF events‘ and subject to the same the fair play rules. That will help organisers who won’t have to formulate their own rules.

Not yet though. The 4NCL published its new fair play guidelines yesterday, to apply in season 2 of the online 4NCL. They're not the same as the ECF guidelines, although I wouldn't be surprised if one document was produced by amending the other.

Two interesting additions the 4NCL has made are:

4NCL Online will manage any changes to results etc in such a way as to avoid or minimise any publicity for a sanctioned player. The presumption will be that previous results will not be changed in the interests of protecting the sanctioned player’s anonymity.

It seems to me that something's not right if you're prepared to ban someone, but not prepared to go public about it. The effectiveness of this seems questionable though - anyone will be able to see the banned player is no longer a member of the squad, and, if also banned by lichess, will be able to see that from their account.

A captain may ask the 4NCL Chief Arbiter to ask if 4NCL Online has any concerns about any players in their squad based on data in its possession as a result of using De Ken Regan’s software. The 4NCL Chief Arbiter may reply in the affirmative or negative only, without revealing the number of players or their identity.

which left me wondering what a captain might do if the answer was yes.

If it was me, I'd be inclined to drop a couple of players from my squad and ask the question again, repeating as many times as was necessary.

[Richard Bates]

I think one big difference between OTB and online chess with regard to cheating is that when playing online they is the constant spontaneous temptation based on opportunity that simply isn’t there in OTB chess, particularly at slower time controls.

Somebody could go into a game with no preconceived intention to cheat, or indeed thought about what method of cheating might be feasible. And play absolutely fair and straight right up until the moment when that critical complicated moment arrives. Success or failure in the match may depend on it. It might even be a fifty:fifty call where any player or any strength may opt for one move or the other.

And at that moment, through some element of weakness deep in their character, they take the risk and access a computer to tell them the right path. There is no chance they will be caught, the moment will pass every casual or probably expert adviser by, and any doubt can always be explained away as “I just rolled the dice”. And there’s no protection beyond an individuals guilty conscience and even that’s not much good after the game.

Whereas in OTB chess a great deal of the anti-cheating work has focussed as much on prevention (denying the means to cheat and at least without drawing attention to unusual behaviour) as detection (based on computer analyses probability). And I would suggest that the former has been far more successful.

I think we’ve had discussions before about how much stronger a player (particularly somebody starting from a high base) might be if given access to a computer for a single move. Online chess simply can’t combat this issue.

I might also note that with OTB chess is has been generally accepted that computer algorithms are only enough to generate suspicion - the method of cheating must also be determined. Whereas to some extent the approach to online chess seems to have become that the method need not be determined (because in most cases it would not be possible to do so) it is enough that the means exists. Which creates an uncomfortable intersection in the Venn diagram of players potentially cheating at chess - those who are ultimately assumed to be innocent OTB in the absence of any discernible means to cheat, but assumed guilty in online chess because it is not realistically possible to create playing conditions to restrict the ability to cheat and push the burden of proof into the organisers.