Bug 1419

Summary: Fix PTY handling on Mac OS X
Product: Portable OpenSSH Reporter: Disco Vince Giffin <vgiffin>
Component: sshAssignee: Assigned to nobody <unassigned-bugs>
Status: CLOSED FIXED    
Severity: normal CC: djm, dtucker
Priority: P2    
Version: 4.7p1   
Hardware: Other   
OS: Mac OS X   
Bug Depends on:    
Bug Blocks: 1481    
Attachments:
Description Flags
Corrects PTY handling.
none
A PTY permission fix.
none
Make pty_release a no-op on OS X
none
pty_release => noop with feature test macro
none
unbroken diff none

Description Disco Vince Giffin 2007-12-21 15:45:00 AEDT
Created attachment 1415 [details]
Corrects PTY handling.

Attached are two patches for building OpenSSH 4.7p1 on Mac OS X.

These patches correct PTY handling on Mac OS X.
Comment 1 Disco Vince Giffin 2007-12-21 15:45:38 AEDT
Created attachment 1416 [details]
A PTY permission fix.
Comment 2 Darren Tucker 2007-12-31 21:56:19 AEDT
Could you please explain why these are needed?  Does ttyname not work on OS X?  And why do you want to not change the ownership of the ptys?
Comment 3 Disco Vince Giffin 2008-01-08 05:14:35 AEDT
(In reply to comment #2)
> Could you please explain why these are needed?  Does ttyname not work
> on OS X?  And why do you want to not change the ownership of the ptys?

I've spoken with the author of these patches and done some testing.

The first patch (for ttyname) is no longer needed.

The second (permission fix) patch bypasses setowner.  We don't need to call setowner because the ownership and permissions are already setup when we allocate a cloning pty.  This patch was added (and is needed) because when thing close down, and the last file descriptor to the pty is closed, the kernel removes it from /dev.  So when sshd (thinking the pty is still there) tries to reset the ownership and permissions, it gets an error because it isn't there anymore.
Comment 4 Darren Tucker 2008-01-08 15:07:59 AEDT
(In reply to comment #3)
> I've spoken with the author of these patches and done some testing.
> 
> The first patch (for ttyname) is no longer needed.

OK then I'll ignore that one.

> The second (permission fix) patch bypasses setowner.  We don't need to
> call setowner because the ownership and permissions are already setup
> when we allocate a cloning pty.

That means pty_setowner is a no-op?  So is the ifdef even required?

There's another call to pty_setowner in monitor.c which will be called when privsep is enabled (the default) so the ifdef around pty_setowner will only have an effect if privsep is off or it's root logging in.

> This patch was added (and is needed)
> because when thing close down, and the last file descriptor to the pty
> is closed, the kernel removes it from /dev.  So when sshd (thinking the
> pty is still there) tries to reset the ownership and permissions, it
> gets an error because it isn't there anymore.

OK that sounds reasonable but it probably belongs in sshpty.c with the rest of the pty handling code.
Comment 5 Darren Tucker 2008-01-08 15:16:30 AEDT
Created attachment 1439 [details]
Make pty_release a no-op on OS X

If my assumptions are correct, this patch should do the right thing on OS X.
Comment 6 Damien Miller 2008-01-20 06:55:14 AEDT
Where/how does __APPLE_PRIVPTY__ set set?
Comment 7 Disco Vince Giffin 2008-01-22 07:52:29 AEDT
(In reply to comment #5)
> Created an attachment (id=1439) [details]
> Make pty_release a no-op on OS X
> 
> If my assumptions are correct, this patch should do the right thing on
> OS X.

This patch does work for Mac OS X (10.5.1).
Comment 8 Disco Vince Giffin 2008-01-22 07:56:01 AEDT
(In reply to comment #6)
> Where/how does __APPLE_PRIVPTY__ set set?

This is set (or not) with an Extra_Configure_Flags in our top level Makefile, depending on what platform we are building for.
Comment 9 Darren Tucker 2008-01-22 15:24:34 AEDT
(In reply to comment #8)
> (In reply to comment #6)
> > Where/how does __APPLE_PRIVPTY__ set set?
> 
> This is set (or not) with an Extra_Configure_Flags in our top level
> Makefile, depending on what platform we are building for.

So that means that people building OpenSSH from source themselves on those platforms will not have the #define and thus get the wrong behaviour?

How can one reliably tell which platforms/versions need that need this?
Comment 10 Disco Vince Giffin 2008-01-23 10:07:12 AEDT
(In reply to comment #9)
> (In reply to comment #8)
> > (In reply to comment #6)
> > > Where/how does __APPLE_PRIVPTY__ set set?
> > 
> > This is set (or not) with an Extra_Configure_Flags in our top level
> > Makefile, depending on what platform we are building for.
> 
> So that means that people building OpenSSH from source themselves on
> those platforms will not have the #define and thus get the wrong
> behaviour?
> 
> How can one reliably tell which platforms/versions need that need this?

Good question.  We could use AvailabilityMacros:

#ifdef __APPLE__
#include <AvailabilityMacros.h>
#if (MAC_OS_X_VERSION_MAX_ALLOWED >= MAC_OS_X_VERSION_10_5)
#define __APPLE_PRIVPTY__
#endif
#endif
Comment 11 Damien Miller 2009-01-21 16:11:52 AEDT
Created attachment 1593 [details]
pty_release => noop with feature test macro

Like this?
Comment 12 Damien Miller 2009-01-21 16:39:28 AEDT
Created attachment 1594 [details]
unbroken diff

oops, that last diff was made of fail.
Comment 13 Damien Miller 2009-02-02 09:22:01 AEDT
(In reply to comment #12)
> Created an attachment (id=1594) [details]
> unbroken diff

Could someone from Apple confirm that attachment 1594 [details] is correct? I'd like to get this in for openssh-5.2.
Comment 14 Damien Miller 2009-02-02 09:22:49 AEDT
(In reply to comment #12)
> Created an attachment (id=1594) [details]
> unbroken diff

Could someone from Apple confirm that attachment 1594 [details] is correct? I'd like to get this in for openssh-5.2.
Comment 15 Disco Vince Giffin 2009-02-03 13:48:22 AEDT
(In reply to comment #14)
> (In reply to comment #12)
> > Created an attachment (id=1594) [details] [details]
> > unbroken diff
> 
> Could someone from Apple confirm that attachment 1594 [details] is correct? I'd
> like to get this in for openssh-5.2.

This works for me (on 10.5.6).
Comment 16 Damien Miller 2009-02-12 12:19:56 AEDT
patch applied - will be in openssh-5.2. Thanks!
Comment 17 Damien Miller 2009-02-23 13:35:41 AEDT
Close bugs fixed/reviewed for openssh-5.2 release