4NCL Online

[Roger Lancaster]

No disrespect to the "strong human players", but some things can be very counterintuitive in probability/stats, and have an understanding or review the data from thousands of human games (which lichess has, and I doubt most strong chess players have).

Each move not being remarkable and having a string of them- may not cause any alarm, but it should.

A famous example from statistics- a professor asks half of his class to flip a coin 200 times and record it, whilst the other half were asked to make it up. Then when they had in the results- he could always tell whether it was made up or not.

The reason? Students making up coin tosses never had 6 heads in a row or 6 tails in a row, when it's virtually impossible for the longest run to be less than 6 in 200 coin tosses.

https://www.csun.edu/~hcmth031/tlroh.pdf

One reason for mentioning Bayes was that, for those unacquainted with probability/statistics, it's counter-intuitive to believe that a 5% error margin results in a 50% probability of a wrong result. But, with due respect to Li Wu, I don't think that students' inability to produce a random string is in any meaningful way comparable to a strong player's [and more so, if more than one] ability to decide whether a series of chess moves was beyond the ability of another known player to play without outside assistance.

[Li Wu]

Was trying to illustrate a point, was probably a bad analogy- i.e. 5 unremarkable moves in a row is not the same as 10 unremarkable moves in a row and not the same as 20 unremarkable moves in a row... but might feel similar for a strong player without looking at the data of thousands of games played.

[Joseph Conlon]

Coming back to the example that has sparked a lot of this discussion, it seems to me that (on appearances) lichess have been entirely reasonable on this.

4NCL is an external league hosting an event on their (free) servers. By all accounts they have offered to explain to the 4NCL board (or something similar) why player X was banned, provided that the 4NCL board make a legal commitment not to make public exactly how their anti-cheating mechanisms work. It is understandable that people involved with the 4NCL do not want to put themselves under any potential legal liability, but in my opinion it is not reasonable to expect that lichess should both be offering a free service to host any 'serious' league games *and* that such leagues have a no-obligation entitlement to see the inner workings of why a player should get a ban. If the 4NCL had paid lichess for the service, that might be a different matter.

Also, to say again: lichess's claimed policy is that one single use of an engine for assistance in a rated game, however slight, will result in a ban. The fact that a strong player doesn't seem anything unusual in the moves is (to me) not that relevant - if there is one single position where stockfish has been turned on, that would result in a ban if detected.

(not making claims either way in respect of the particular example though)

[MartinCarpenter]

99.99% accuracy is nothing like enough, I think. A false ban every 10,000 games?

In general, it seems fair to assume that a cheater won't stop cheating, so you just flag people that trigger something once and wait. All their later games are independent events, so it won't take terribly long before the accuracy is off the charts.

Much more challenging with one off events and prize money of course.

[Roger Lancaster]

Was trying to illustrate a point, was probably a bad analogy- i.e. 5 unremarkable moves in a row is not the same as 10 unremarkable moves in a row and not the same as 20 unremarkable moves in a row... but might feel similar for a strong player without looking at the data of thousands of games played.

Point made, and apologies if I sounded critical. The other point which occurs to me is this. If you had a system which worked 99.99% of the time, and we seem to be saying that's the order of accuracy required here with the suggestion [consistent with what Thibault says in a YouTube video mentioned elsewhere] that it's multi-layered, would you still refuse - even when it became apparent that not everyone believed in this degree of accuracy - to disclose any details on the grounds that it would assist those trying to get around the system?

My turn to try an analogy. If I had patented a new lock which I knew would thwart the lock-breaking efforts of 99.99% of thieves and burglars, would I seriously say, "This lock will defy virtually all lock-pickers but, if I told you one word more, it would help those trying to break in"? Obviously not, because few would believe me. Instead, I'd give enough information to instill confidence in my product without divulging the really sensitive stuff.

[JustinHorton]

I think your reasoning is sound, and I do think the sites know this well and that they really go for "beyond reasonable doubt".

There is no reason to believe this, and this...

The reason I'm sceptical about 99.99% is the number of cases - Northumbria springs to mind as a likely example - where games are analysed by strong human players who can find nothing suspicious.

...is a good reason not to.

No disrespect to the "strong human players", but some things can be very counterintuitive in probability/stats, and have an understanding or review the data from thousands of human games (which lichess has, and I doubt most strong chess players have).

No, this won't do at all. You can't have a situation where the evidence of your own well-informed eyes is to be discounted because the statistics supposedly say something else. There's a few reasons for this but one is, how do you know the statistics are reliable in the first place? And how do you know they're being properly interpreted?

All these type of arguments invite us to make assumptions about procedures that we're simply not entitled to make, which we simply don't have the information in front of us that would enable us to consider valid.

[JustinHorton]

Really I don't want to make everything about myself, but my experience does serve as a pretty good example, precisely because there's really not that many games to consider, and there's absolutely no way they could be interpreted to show cheating "beyond reasonable doubt" or anything close to it. There's no conceivable way that any kind of scrutiny even of the most cursory nature could show this, and the most generous interpretation available is that a number of assumptions were made about my playing strength and my access to opening theory that were simply invalid. Even then there are so many question marks attached that it is simply not plausible that a proper process was undertaken unless we are going to assume that it was regardless of all other evidence.

Their stats were bad, their assumptions were bad, their procedures were bad.

[Roger de Coverly]

The fact that a strong player doesn't seem anything unusual in the moves is (to me) not that relevant - if there is one single position where stockfish has been turned on, that would result in a ban if detected.

But how are you doing the "if detected" bit? Fine if it's a normal 4NCL weekend and someone observed the player looking at a computer, tablet or phone. That could be loss of game at the very least on FIDE laws of chess.

But if the detection relies on the theory that
(i)an engine has recommended a series of strong moves
(ii) player has found the same strong moves but isn't according to his rating strong enough to find them
(iii) then he must have consulted an engine
that isn't credible.

It's even less credible if it's a single position where use is claimed to have been detected.

[MartinCarpenter]

The stats are OK. Not perfect, but little is.

You take a large population of human games played with no computer assistance (historical if nothing else), you detect features in play that weren't seen then and you can use that to stick a probability on the chance of a game being played by a human vs with an engine.

The idea is fundamentally sound enough, and they've refined it a few ways by now.

My only genuine worry with them would be what happens if you catch someone in very deep preparation. That's actually very plausible in something like the 4NCL online - some UK players are near enough sitting ducks if you've got a great memory and the time/motivation.

That's very hard to screen out and could definitely entirely mess the stats up.

[Joseph Conlon]

But how are you doing the "if detected" bit? Fine if it's a normal 4NCL weekend and someone observed the player looking at a computer, tablet or phone. That could be loss of game at the very least on FIDE laws of chess.

But if the detection relies on the theory that
(i)an engine has recommended a series of strong moves
(ii) player has found the same strong moves but isn't according to his rating strong enough to find them
(iii) then he must have consulted an engine
that isn't credible.

It's even less credible if it's a single position where use is claimed to have been detected.

Roger, this I absolutely agree on - that simple correlations with engine preferred moves by itself would be the weakest grounds on which to ban someone.

The sort of stuff I am thinking about is where there is some detection of what else is going on on your computer. I don't know what is possible and what is not, but let me give hypothetical examples.

(1) If someone is playing on lichess with another open lichess analysis tab with their game position on, then clearly lichess can bust them instantly.

(2) Suppose was playing on lichess, with a chess.com analysis tab open with their game position. Now, *IF* there was some data sharing between lichess and chess.com on ip addresses, game positions and open analyses, then they could be busted by lichess.

Now, I've no idea whether (2) does happen (or whether it would be legal). But if it does, I imagine they wouldn't want to divulge it. And if there was, say, such sharing between lichess and chess.com, but not with chess24, then I can imagine even more that they would not want to divulge it.

[Roger de Coverly]

You take a large population of human games played with no computer assistance (historical if nothing else), you detect features in play that weren't seen then and you can use that to stick a probability on the chance of a game being played by a human vs with an engine.

Having developed a witchfinder program, if it's tested on games known to be clean, then how many false detections does it make? Over time from say around the early to mid 1990s I would expect the correlation with engine moves to increase particularly in early middle games. Not because of undetected cheating, but because players train with engines and take notice of their suggestions.

[Roger de Coverly]

(1) If someone is playing on lichess with another open lichess analysis tab with their game position on, then clearly lichess can bust them instantly.

It's occurred to me that should be easy enough to circumvent and the big problem is allowing analysis in real time of games in progress. That's fine for over the board tournaments where the nature of play makes live analysis of the games relatively secure from the players being able to access it and a limited number of games have live coverage.

[Li Wu]

@Justin reading some of what I wrote I realise that it was more of my biases talking- as I don't know the inner workings of lichess (or chess.com for that matter) I'm putting quite a bit of faith in their methodology. Still, I trust them (at least Bayesian inference is a much simpler topic than what they have been working on).

Having developed a witchfinder program, if it's tested on games known to be clean, then how many false detections does it make? Over time from say around the early to mid 1990s I would expect the correlation with engine moves to increase particularly in early middle games. Not because of undetected cheating, but because players train with engines and take notice of their suggestions.

To come back to this point- this has been shown to be false of sorts - at least cheat detection can take this into account. It seems that ratings have maintained to be very good predictors of playing strength throughout the ages, and the loose statement:

"current players have better correlation with engine moves" is basically equivalent to "current players have higher elo".

A 2100 player today has more or less no better engine correlation than a 2100 player in the 90s- the only serious difference being opening theory being different. It's true that computers (engines widespread availability of chess theory/knowledge in electronic form) has produced better and better players- but this is reflected in their higher ratings and quicker improvements by juniors.

https://cse.buffalo.edu/~regan/papers/pdf/ReHa11c.pdf - paper with data on "myth of rating inflation"

[Li Wu]

Also @Joseph- this is a big reason I think why they are so reluctant to reveal anything about their detection methods.

Categorically saying they have something, or nothing, even without pointing out what it is, can help cheaters.

[JustinHorton]

It can also help them cover up their own mistakes.