Bug 3530 - AC_CHECK_FUNCS of libfido2 functions generate wrong results if not using pkg-config
Summary: AC_CHECK_FUNCS of libfido2 functions generate wrong results if not using pkg-...
Status: CLOSED FIXED
Alias: None
Product: Portable OpenSSH
Classification: Unclassified
Component: Build system (show other bugs)
Version: 9.1p1
Hardware: All Linux
: P5 normal
Assignee: Damien Miller
URL:
Keywords:
Depends on:
Blocks: V_9_2
  Show dependency tree / graph
 
Reported: 2023-02-01 06:43 AEDT by yzhangad
Modified: 2023-03-17 13:38 AEDT (History)
2 users (show)

See Also:

Attachments
Implement James' fix (429 bytes, patch)
2023-02-01 09:23 AEDT, Damien Miller 		dtucker: ok+
Details | Diff
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description yzhangad 2023-02-01 06:43:57 AEDT 
When autoconf uses `AC_CHECK_FUNCS` to test the existence of necessary libfido2 functions, it appends `$LIBFIDO2` to `$LIBS`: https://github.com/openssh/openssh-portable/blob/a20b7e999773e6333c8aa9b0a7fa41966e63b037/configure.ac#L3248. In an environment without `pkg-config`, this makes `-lfido2 -lcbor` appear at the end of the linker flags.

The problem of this: `libcbor` is not the only dependency of `libfido2`: it also depends on, e.g. `libcrypto`, which is part of OpenSSL, and that can be configured via the `--with-ssl-dir` option. When OpenSSL and `libfido2` are built statically, the existing `configure.ac` makes `-lcrypto` go before `-lfido2`, making the linkage break, thus causing false negative results of `AC_CHECK_FUNCS`.

The fix should consist just of replacing `LIBS="$LIBS $LIBFIDO2"` with `LIBS="$LIBFIDO2 $LIBS"`.
Comment 1 Damien Miller 2023-02-01 09:23:59 AEDT 
Created attachment 3654 [details]
Implement James' fix
Comment 2 Damien Miller 2023-02-01 12:18:21 AEDT 
Applied and will be in the OpenSSH 9.2 release due shortly.
Comment 3 Damien Miller 2023-03-17 13:38:51 AEDT 
OpenSSH 9.3 has been released. Close resolved bugs