I have been experimenting with the ChrootDirectory feature on OpenSSH 5.2p1 running on AIX 5.3, but have encountered repeated failures
chroot("/restrict/home"): Operation not permitted.

Debugging the sshd process, it is the "chroot()" subroutine failing with error EPERM "Operation not permitted" because the process no longer has root user authority. Under AIX, the manual pages for the "chroot()" subroutine say "The calling process must have root user authority in order to change the effective root directory."

I believe the problematic code is located in the "session.c" module within function "do_setusercontext()". In this function, the "setpcred()" subroutine is called to change the user/group privileges from the root user to that of the ssh user. This is later followed by "safely_chroot()" (which invokes "chroot()"). Unfortunately the order of these calls wont work for non-root users on AIX. To make it work, the "safely_chroot()" must be called before "setprcred()".

Solaris has a similar restriction for "chroot()", viz: "The {PRIV_PROC_CHROOT} privilege is not asserted in the effective set of the calling process."