Bug 1419 - Fix PTY handling on Mac OS X
Summary: Fix PTY handling on Mac OS X
Status: CLOSED FIXED
Alias: None
Product: Portable OpenSSH
Classification: Unclassified
Component: ssh (show other bugs)
Version: 4.7p1
Hardware: Other Mac OS X
: P2 normal
Assignee: Assigned to nobody
URL:
Keywords:
Depends on:
Blocks: V_5_2
  Show dependency treegraph
 
Reported: 2007-12-21 15:45 AEDT by Disco Vince Giffin
Modified: 2023-01-13 13:56 AEDT (History)
2 users (show)

See Also:


Attachments
Corrects PTY handling. (792 bytes, patch)
2007-12-21 15:45 AEDT, Disco Vince Giffin
no flags Details | Diff
A PTY permission fix. (763 bytes, patch)
2007-12-21 15:45 AEDT, Disco Vince Giffin
no flags Details | Diff
Make pty_release a no-op on OS X (589 bytes, patch)
2008-01-08 15:16 AEDT, Darren Tucker
no flags Details | Diff
pty_release => noop with feature test macro (1.10 KB, patch)
2009-01-21 16:11 AEDT, Damien Miller
no flags Details | Diff
unbroken diff (1.05 KB, patch)
2009-01-21 16:39 AEDT, Damien Miller
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Disco Vince Giffin 2007-12-21 15:45:00 AEDT
Created attachment 1415 [details]
Corrects PTY handling.

Attached are two patches for building OpenSSH 4.7p1 on Mac OS X.

These patches correct PTY handling on Mac OS X.
Comment 1 Disco Vince Giffin 2007-12-21 15:45:38 AEDT
Created attachment 1416 [details]
A PTY permission fix.
Comment 2 Darren Tucker 2007-12-31 21:56:19 AEDT
Could you please explain why these are needed?  Does ttyname not work on OS X?  And why do you want to not change the ownership of the ptys?
Comment 3 Disco Vince Giffin 2008-01-08 05:14:35 AEDT
(In reply to comment #2)
> Could you please explain why these are needed?  Does ttyname not work
> on OS X?  And why do you want to not change the ownership of the ptys?

I've spoken with the author of these patches and done some testing.

The first patch (for ttyname) is no longer needed.

The second (permission fix) patch bypasses setowner.  We don't need to call setowner because the ownership and permissions are already setup when we allocate a cloning pty.  This patch was added (and is needed) because when thing close down, and the last file descriptor to the pty is closed, the kernel removes it from /dev.  So when sshd (thinking the pty is still there) tries to reset the ownership and permissions, it gets an error because it isn't there anymore.
Comment 4 Darren Tucker 2008-01-08 15:07:59 AEDT
(In reply to comment #3)
> I've spoken with the author of these patches and done some testing.
> 
> The first patch (for ttyname) is no longer needed.

OK then I'll ignore that one.

> The second (permission fix) patch bypasses setowner.  We don't need to
> call setowner because the ownership and permissions are already setup
> when we allocate a cloning pty.

That means pty_setowner is a no-op?  So is the ifdef even required?

There's another call to pty_setowner in monitor.c which will be called when privsep is enabled (the default) so the ifdef around pty_setowner will only have an effect if privsep is off or it's root logging in.

> This patch was added (and is needed)
> because when thing close down, and the last file descriptor to the pty
> is closed, the kernel removes it from /dev.  So when sshd (thinking the
> pty is still there) tries to reset the ownership and permissions, it
> gets an error because it isn't there anymore.

OK that sounds reasonable but it probably belongs in sshpty.c with the rest of the pty handling code.
Comment 5 Darren Tucker 2008-01-08 15:16:30 AEDT
Created attachment 1439 [details]
Make pty_release a no-op on OS X

If my assumptions are correct, this patch should do the right thing on OS X.
Comment 6 Damien Miller 2008-01-20 06:55:14 AEDT
Where/how does __APPLE_PRIVPTY__ set set?
Comment 7 Disco Vince Giffin 2008-01-22 07:52:29 AEDT
(In reply to comment #5)
> Created an attachment (id=1439) [details]
> Make pty_release a no-op on OS X
> 
> If my assumptions are correct, this patch should do the right thing on
> OS X.

This patch does work for Mac OS X (10.5.1).
Comment 8 Disco Vince Giffin 2008-01-22 07:56:01 AEDT
(In reply to comment #6)
> Where/how does __APPLE_PRIVPTY__ set set?

This is set (or not) with an Extra_Configure_Flags in our top level Makefile, depending on what platform we are building for.
Comment 9 Darren Tucker 2008-01-22 15:24:34 AEDT
(In reply to comment #8)
> (In reply to comment #6)
> > Where/how does __APPLE_PRIVPTY__ set set?
> 
> This is set (or not) with an Extra_Configure_Flags in our top level
> Makefile, depending on what platform we are building for.

So that means that people building OpenSSH from source themselves on those platforms will not have the #define and thus get the wrong behaviour?

How can one reliably tell which platforms/versions need that need this?
Comment 10 Disco Vince Giffin 2008-01-23 10:07:12 AEDT
(In reply to comment #9)
> (In reply to comment #8)
> > (In reply to comment #6)
> > > Where/how does __APPLE_PRIVPTY__ set set?
> > 
> > This is set (or not) with an Extra_Configure_Flags in our top level
> > Makefile, depending on what platform we are building for.
> 
> So that means that people building OpenSSH from source themselves on
> those platforms will not have the #define and thus get the wrong
> behaviour?
> 
> How can one reliably tell which platforms/versions need that need this?

Good question.  We could use AvailabilityMacros:

#ifdef __APPLE__
#include <AvailabilityMacros.h>
#if (MAC_OS_X_VERSION_MAX_ALLOWED >= MAC_OS_X_VERSION_10_5)
#define __APPLE_PRIVPTY__
#endif
#endif
Comment 11 Damien Miller 2009-01-21 16:11:52 AEDT
Created attachment 1593 [details]
pty_release => noop with feature test macro

Like this?
Comment 12 Damien Miller 2009-01-21 16:39:28 AEDT
Created attachment 1594 [details]
unbroken diff

oops, that last diff was made of fail.
Comment 13 Damien Miller 2009-02-02 09:22:01 AEDT
(In reply to comment #12)
> Created an attachment (id=1594) [details]
> unbroken diff

Could someone from Apple confirm that attachment 1594 [details] is correct? I'd like to get this in for openssh-5.2.
Comment 14 Damien Miller 2009-02-02 09:22:49 AEDT
(In reply to comment #12)
> Created an attachment (id=1594) [details]
> unbroken diff

Could someone from Apple confirm that attachment 1594 [details] is correct? I'd like to get this in for openssh-5.2.
Comment 15 Disco Vince Giffin 2009-02-03 13:48:22 AEDT
(In reply to comment #14)
> (In reply to comment #12)
> > Created an attachment (id=1594) [details] [details]
> > unbroken diff
> 
> Could someone from Apple confirm that attachment 1594 [details] is correct? I'd
> like to get this in for openssh-5.2.

This works for me (on 10.5.6).
Comment 16 Damien Miller 2009-02-12 12:19:56 AEDT
patch applied - will be in openssh-5.2. Thanks!
Comment 17 Damien Miller 2009-02-23 13:35:41 AEDT
Close bugs fixed/reviewed for openssh-5.2 release