Created attachment 3061 [details] patch In the past couple of years the Linux kernel gained support for VRF. Applications can bind to a specific VRF via the SO_BINDTODEVICE socket option. Add a new -B option that takes a string as a parameter to both ssh and sshd, and use it to bind the socket. https://www.kernel.org/doc/Documentation/networking/vrf.txt Original mailing list thread asking for this feature: https://lists.mindrot.org/pipermail/openssh-unix-dev/2015-November/034525.html A patch implementing the feature is attached. Tested by creating a VRF and enslaving a virtual device to it, that connects to a VM. Tested that the VM is reachable over the VRF, and not "normally".
With VRF on Linux systems network addresses need to be qualified by a VRF. We have been recommending '%' as the delimiter for that. For example the ListenAddress directive for sshd could be something like <addr>%<vrf>. Generically for sshd that could be <addr>%<dev> in which case sshd binds sockets first to the device and then to the address.
I don't think % is a great choice because it is already used for interface-scoped IPv6 addresses. A separate keyword extending ListenAddress might be a better choice, e.g. ListenAddress 0.0.0.0:22 domain foo-vrf or somesuch. I'll have to survey the other free OS's to see what they implement so we can try make it general enough for all of them. OpenBSD at least has routing domains that seem comparable, and I'd be surprised if FreeBSD lacked a similar feature.
Created attachment 3064 [details] patch v2 Hi, As suggested, v2 of the patch adds a "domain" keyword to the ListenAddress option. The -B or BindDevice option, like the -p and Port ones, offer a default global value, that individual ListenAddress domains can override. I had to do a small refactor on how the ListenAddress addresses are stored, since a single hostname can in theory have multiple IP addresses, which must all be associated with the same domain. Previously the struct addrinfo linked lists were used directly, now they are wrapped in a struct that also has a pointer to the domain. I do not have a preference on using "%vrf" vs "domain vrf". It would be nice to have consistency across different programs, but it is also true that linklocal addresses already use it. Up to you! I'm happy to test either. It's the first time I work on this code base so please double, nay triple check :-) Thanks!
I really don't have a strong preference of '%' versus 'domain' but I do want to clarify one thing: <addr>%<vrf> has the same intent as a linklocal address which is <addr>%<dev> -- both limit the scope of the address to a device (on Linux, VRF is implemented using a net_device so device based APIs work with it). Either way ('%' or 'domain') I would love to see this feature supported by openssh
Created attachment 3070 [details] Add rdomain option to ListenAddress I'm implemented this a little differently, starting on OpenBSD since that's where the patch would have to land first. This is the first piece - adding an rdomain option to ListenAddress, e.g. ListenAddress 0.0.0.0 ListenAddress 0.0.0.0 rdomain 4
Created attachment 3071 [details] Add RoutingDomain option This patch adds a RoutingDomain option to place sessions into the specified routing domain after authentication completes. E.g. Match user !root,* RoutingDomain 4
Oh, and it's possible to "copy" the routing domain from the incoming connection too using: RoutingDomain %D
Created attachment 3072 [details] add rdomain match criteria This adds an rdomain criteria to the sshd_config Match keyword, e.g. Match rdomain 4 PasswordAuthentication no Banner /etc/public-disclaimer
FYI OpenBSD routing domains are identified by integers [0, 255], but my patches handle them as strings except for the actual syscalls to get/set them, to make life easy for portable OpenSSH. We should only need to replace four functions: misc.c:get_rdomain() misc.c:set_rdomain() servconf.c:valid_rdomain() sshd.c:set_process_rdomain()
Created attachment 3075 [details] ListenAddress rdomain qualifier Revised ListenAddress rdomain qualifier, with fewer bugs and more documentation.
Created attachment 3076 [details] add RDomain option Revised RDomain (was RoutingDomain) option for sshd_config; with more documentation
Created attachment 3077 [details] rdomain Match criteria Revised rdomain sshd_config match criteria. With documentation
get/set_rdomain can be implemented on Linux using SO_BINDTODEVICE, but note the caveat that applies to get_rdomain: > Before Linux 3.8, this socket option could be set, but could not > retrieved with getsockopt(2). Since Linux 3.8, it is readable. (from https://linux.die.net/man/7/socket) Implementing set_process_rdomain() on Linux seems a little convoluted, but "ip vrf exec" shows the procedure (NB. we can't copy the code because of licensing) https://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git/tree/ip/ipvrf.c?id=4b73d52f8a81919f511cd47d39251f74f6a37c7d#n353
The API for Linux is SO_BINDTODEVICE. 'ip vrf exec' is a helper for applications that do not natively support SO_BINDTODEVICE (or similar APIs). The helper is not applicable for sshd; the intent is to support the API directly. VRF support was added in v4.3, so any attempts to use SO_BINDTODEVICE for VRF will be running a newer kernel. Finally, should any code be needed from Linux I wrote both the Linux kernel implementation and the vrf support in iproute2.
I think one of us is misunderstanding; how does one bind a process tree to a VRF using SO_BINDTODEVICE? That's what the RDomain keyword does...
I am thinking about this in terms of sshd and a listen socket bound to a VRF or an ssh client opening a socket contained in a VRF. I am not familiar with openssh and process tree use cases.
Right now there are no use cases, these patches add them for the first time. The functionality in question here is: 1. Being able to tell sshd to listen in an explicit rdomain/VRF. This is the first patch, implementing ListenAddress addr[:port] [rdomain domain] This seems like SO_BINDTODEVICE will work fine. 2. Being able to set the rdomain/VRF for sshd, so the user session as well as any sockets created for forwardings end up in an rdomain. This is the second patch, implementing RDomain domain I can't see how SO_BINDTODEVICE will work here, because it won't affect sshd's child processes (e.g. the user's shell). OpenBSD provides a setrtable(2) syscall to do this that has sensible semantics: https://man.openbsd.org/setrtable.2
Ok, I see. I wrote 'ip vrf exec' and its kernel side implementation using cgroups for case 2. This is not something we intended to be done from sshd.
Are you saying that it's an unsupported interface or that we should otherwise avoid using it?
Hi Damien, On what git tree are the 3 patches to be applied on? git am fails on the current openssh-portable master branch, and on the latest tag as well. I wanted to add the Linux-specific bits and test them. Thanks!
The patches are against OpenBSD. Once I've got them committed upstream I was planning on starting implementing the Linux bits - probably by extending openbsd-compat/port-tun.[ch] to contain VRF stuff too and renaming it it to port-net.[ch] in the process.
Damien: It is a supported interface. In fact someone else is looking at adding the capability to systemd. My point is that 'ip vrf exec' and process tree style control does not seem appropriate for sshd. If there is a use case for it then it can be added. The key is a v2 cgroup and applying a BPF filter. Also, you need a v4.10 and higher kernel.
Created attachment 3078 [details] rdomain support for ssh client This patch applies similar changes to the ssh client as the previous patches to sshd, and adds support for BSD routing domain via a RDomain ssh_config option and a new -r command line option. The changes are documented in the respective manpages.
Created attachment 3079 [details] linux vrf support for ssh client This patch builds on top of the changes to support routing domain in the ssh client, and adds the required ifdef'd branches to apply the same config file/command line option as a Linux vrf. Tested with Linux/iproute2 4.9.
Thanks Damien. I've added support for RD and VRF to the ssh client as well in 2 separate patches, tested on Linux, in case it can be useful. w.r.t. the support for forking processes IMHO it can wait a bit and doesn't need to block the rest as long as what works is well documented - as far as I know the main use cases we need for our customers are covered by support for the listen socket in sshd and the connecting socket in ssh. I imagine the same is true for David at Cumulus. Thanks!
I don't think there is any need for rdomain support in the client. "route exec" exists on OpenBSD and "ip vrf exec" on Linux that achieve exactly the same result. We actually had similar support in ssh previously and removed it for this reason. Rdomain/VRF support in the server is different, because there we are adding capabilities that can't be replicated using other tools (listening in arbitrary domains, placing user sessions in a domain, using domain on which a connection was received as a Match criteria)
On Linux "ip vrf exec" unfortunately requires using cgroups V2 on the system, which as of today latest RC candidates is still missing important features, so not everyone can use it. For example it does not implement a CPU controller, which is necessary to effectively partition the CPUs on a system. This is a real use case we have in production with the Vyatta vRouter at AT&T - unfortunately we cannot use cgroups v2 and thus we cannot use "ip vrf exec". Like our case, there will be many more out there. Another example are the Long Term Support kernels like 4.4 that are going to be maintained for 6 years - they either lack the command altogether or cgroups v2 is even more limited. For these reasons IMHO it would be worth to add client-side support in openssh-portable for the Linux case. I'm happy to rework the 2 patches and test them again if you like. Thanks!
I'm reluctant to add rdomain support to the client because, to do it properly, it involves touching every socket() call that it makes. Your patch just binds the connection socket, but all subsequent forwarding sockets will be in the default vrf. BTW, the upstream patches are committed and I've placed stubs in openbsd-compat/port-net.[ch] for the functions that need replacing.
the get/set_rdomain functions seem pretty easy to write, they are just SO_BINDTODEVICE, but I'm not sure how to write valid_rdomain() - is there an easy way to determine whether an interface name exists and represents a VRF under Linux?
I believe the only way to verify the device kind is an rtnetlink message, RTM_GETLINK. In the response look for the IFLA_INFO_KIND attribute.
That looks more fiddly than I have time for this week, so I've added a minimal valid_rdomain() implemention that just tries to SO_BINDTOADDR the supplied name. This will likely accept some non-VRF interface names, but will do as a first approximation until someone writes the rtnetlink glue to do it properly.
Created attachment 3080 [details] rdomain support for ssh client connect socket This new patch builds on top of the changes recently committed in master, to add RDomain support to the ssh client connecting socket (forwarding sockets are dealt with in the following patches).
Created attachment 3081 [details] rdomain support for ssh client local-forward sockets Add support for binding the local forwarding socket to a routing domain.
Created attachment 3082 [details] rdomain support for ssh client remote-forward socket Add support for connecting to the local application port SSH is forwarding traffic to via a Routing Domain. Like for the LocalForward, a 5th parameter is added to the tuple.
Hi Damien, First of all, thank you very much for taking care of the sshd support upstream. I've tested it on Linux, and I can confirm it works perfectly. And thanks for the feedback as well - it's true that I didn't touch the other sockets. I've now attached 3 proposed patches for the ssh client, that add support for the forwarding sockets as well. Among other things, this allows for example to listen for traffic on RD foo and forward it on RD bar.
Hi Damien - did you have any chance to have a look at the client patches? Thanks!
Hi Luca, IMO those client-side patches are a bit too intrusive given that "ip vrf exec" will make them unnecessary in the future once cgroups v2 is stable and full-featured.
> once cgroups v2 is stable and full-featured. I admire your optimism! :-D Yes I agree that it's a bit invasive as there are multiple sockets and features to account for, and for the normal use case ip exec will suffice when cgoups2 is complete (hope to see that before I retire :-P ). There are other corner cases where, unless I'm mistaken, I think that won't be enough though - for example, doing forwarding over a vrf and connecting via another, eg: ssh -r vrf-blue -L 192.168.122.1:9001:localhost:22226 192.168.122.1 or ssh -L 192.168.122.1:9001:vrf-blue:localhost:22226 192.168.122.1 Since ip exec is an "all or nothing" deal (I'm not sure if there is a BSD equivalent). Do you know of a way to achieve the same functionality without changing the client? Now strictly speaking I don't need these cases at $work so I'm not too fussed about it, but some other users might want that at some point so I thought it would be worth at least mentioning. Thanks!
It would be nice to have this feature added. There is no way to tunnel across VRF since vrf exec would bind to a single vrf. I'm hoping for this to work: ssh -L 8888:172.16.0.2:22:vrf-blue 192.168.1.1 In other words, on 192.168.1.1, there is a virtual router vrf-blue that holds the 172.16.0.0/24 network, and I want to tunnel onto that network from a non-vrf port (192.168.1.1).