When a link-local address is used with ListenAddress, the resulting bind() will fail with error as per the following log entry:
error: Bind to port 22 on fe80::202:5555:fe01:dddd failed: Invalid argument.

When using link-local, the sin6_scope_id must be set to specify which interface should be used with the link-local address.  There does not appear to be any handling for this in server_listen() function.

I believe the ListenAddress should allow a link-local with %<zone_id> allowing the interface name or index to be specified as per RFC4007.  The specified interface needs to then result in a sin6_scope_id assignment prior to the bind() in server_listen().  (Since the inclusion of an interface name would result in longer strings, the code needs to be checked that longer addresses with ListenAddress will be accepted.)  The if_nametoindex() function may be needed to convert an interface name into its respective index which gets used as the sin6_scope_id value.