The daemon calls get_remote_port() in auth_log(). If the client closes the
connection between the authentication failure and the invocation of auth_log()
(e.g. during PAM-enforced post-failure delay) then getpeername() fails (the
socket is not connected any longer), get_sock_port() kills the daemon (almost
silently only a debugging message!), and auth_log() is never finished.

One possible fix is to make get_remote_port() cache information in the same way
it is cached by get_remote_ipaddr().