ssh-add / ssh-agent should have an option to start counting private key expiry time from the moment it was last time used. That way, one would only need to re-enter the private key's password if he actually left the desk for a prolonged time rather than once in constant period of time.
*** Bug 496 has been marked as a duplicate of this bug. ***
Closing: Nobody has stepped up to implement this in the last eight years and I have no plans to do so.
Mass move of bugs RESOLVED->CLOSED following the release of openssh-5.5p1