Stockfish 15.1

[Ian Thompson]

Stockfish 15.1 was released a couple of days ago.

Their website contains the interesting comment that:

This release also introduces a new convention for the evaluation that is reported by search. An evaluation of +1 is now no longer tied to the value of one pawn, but to the likelihood of winning the game. With a +1 evaluation, Stockfish has now a 50% chance of winning the game against an equally strong opponent.

That raises a few questions, such as what does an evaluation of +0.5, +2, or anything else mean as a percentage chance of winning the game? Presumably, it won't work properly when used as an engine with any software that expects evaluations it gets from the engine to be related to pawn advantages.

I can't find anything more on it than what's on the Stockfish website and a tweet from Matthew Sadler.

Anyone else found more information about it?

[Roger de Coverly]

That raises a few questions, such as what does an evaluation of +0.5, +2, or anything else mean as a percentage chance of winning the game?

The other known point is that +0.0 equates to a belief by Stockfish that it can draw against itself. But is their distribution linear? Presumably not, or it would be capped at +2 when Stockfish believes it can beat itself with 100% reliability.

[NickFaulks]

"With a +1 evaluation, Stockfish has now a 50% chance of winning the game against an equally strong opponent."

And a 0% chance of losing? That is presumably implied, but must be wrong in general.

[Roger de Coverly]

And a 0% chance of losing? That is presumably implied, but must be wrong in general.

They are using the reporting method first seen with Alpha Zero, namely the number of wins or percentage from a Monte Carlo process. Or at least I think that's how it works. Have the statistics of the number of draws and losses ever been reported in a similar manner? Perhaps they ought to express it terms of a 100 game match, so that plus 1 indicates a 75% result (50 wins 50 draws) or whatever.

[Brian Egdell]

Instead of thinking of the probability of winning against an equally strong opponent, think of the probability of not winning. I.E. drawing or losing. This is apparently also 50% when the advantage is +1. Then think of a factor K to multiply that probability by whenever you add one more point to that advantage.

For example, K might be two thirds.

That would mean that an evaluation of 0.0 (equal position) means that you have a probability of 75% of not winning the game against an opponent of equal strength. (That would evidently mean 25% probability of winning, 25% of losing, and 50% of a draw.)

Adding one to the evaluation, giving 1.0, multiplies the 75% probability of not winning against equal strength opposition by two thirds to give 50%.

Adding one again to give 2.0 evaluation means 50% * 2//3 = 33.3% probability of not winning.

Et cetera.

I don't know what the value of K actually is in this construction, but I am guessing that it works along these lines.

Maybe the change has been done because the value of a pawn is a little too arbitrary and rigid for chess positions as evaluated by modern programs.

[MartinCarpenter]

"With a +1 evaluation, Stockfish has now a 50% chance of winning the game against an equally strong opponent."

And a 0% chance of losing? That is presumably implied, but must be wrong in general.

The chance of SF 15 losing from a slightly better position isn't precisely zero but is certainly frighteningly low. It just doesn't do any of the human things that make us lose them.

[Angus French]

lichess apparently now uses Stockfish 15.1 for whole-game analysis but 14+ NNUE for examination of individual moves. I'm guessing the evaluation scheme is the same for both but can't see that it has been explained.

[MartinCarpenter]

lichess apparently now uses Stockfish 15.1 for whole-game analysis but 14+ NNUE for examination of individual moves. I'm guessing the evaluation scheme is the same for both but can't see that it has been explained.

Should be yes, the evaluation functions are all (I'm fairly sure) based on neural nets trained by massive numbers of self played games. So a %age expected score is actually what the numbers do mean these days.

Of course the score that SF gets playing against itself is not automatically a reliable indicator of how well humans will score from a given position! You could probably actually use databases and so on to try and train something that predicted the expected human %age score.

[Roger de Coverly]

Of course the score that SF gets playing against itself is not automatically a reliable indicator of how well humans will score from a given position! You could probably actually use databases and so on to try and train something that predicted the expected human %age score.

It would be useful to humans to have some measure of the volatility of the advantage. You get Stockfish evaluations of plus 2 or plus 3 with level material. That means the opponent is believed to be busted, but some degree of tactical accuracy is needed to maintain the advantage. That can contrast with say just being a pawn up where there's nothing really happening. The task then being to nurse the extra material to victory.