Skip the initial empty-password check if permit_empty_passwd is disabled. This doesn't change the timing profiles of the host because the additional condition check which can short-circuit the call to pam_authenticate() has no dependency on the identity of the user who is being authenticated.
Created attachment 1684 [details] Patch solving the problem
Seems reasonable, adding to the list for 5.4
Patch applied - thanks.
Move resolved bugs to CLOSED after 5.7 release