openssh_RSA_verify in ssh-rsa.c defines ret to be of type size_t, which is unsigned. It then assigns signed values such as SSH_ERR_INTERNAL_ERROR (-1) to ret. Finally, it returns ret (a size_t), while the method signature of openssh_RSA_verify is defined as returning type int. The method works as intended to some degree, in that on success it returns 0 and on failure it returns non-zero. But if one were to try to do something with the return value on failure, one would find it to be garbage. The same goes for trying to observe the return value with a debugger. This problem could easily be fixed by declaring ret to be of type int.