Bug 1239 - OpenSSL ENGINE support broken on Solaris using Sun Studio compiler
Summary: OpenSSL ENGINE support broken on Solaris using Sun Studio compiler
Status: CLOSED FIXED
Alias: None
Product: Portable OpenSSH
Classification: Unclassified
Component: Build system (show other bugs)
Version: 4.4p1
Hardware: Other Solaris
: P2 normal
Assignee: Assigned to nobody
URL:
Keywords:
Depends on:
Blocks: V_4_5
  Show dependency treegraph
 
Reported: 2006-09-29 00:04 AEST by Andrew Benham
Modified: 2008-04-04 09:56 AEDT (History)
0 users

See Also:


Attachments
Fix for configure.ac (581 bytes, patch)
2006-09-29 00:07 AEST, Andrew Benham
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Andrew Benham 2006-09-29 00:04:40 AEST
Solaris 8,9, and 10, x86 and sparc

configuring with '--with-ssl-engine' gives an error:

checking for OpenSSL ENGINE support... configure: error: OpenSSL ENGINE support not found


Cause:
configure.ac lines 1916 onward state:

                AC_TRY_COMPILE(
                        [ #include <openssl/engine.h>],
                        [
int main(void){ENGINE_load_builtin_engines();ENGINE_register_all_complete();}
                        ],
                        [ AC_MSG_RESULT(yes)
                          AC_DEFINE(USE_OPENSSL_ENGINE, 1,
                             [Enable OpenSSL engine support])
                        ],
                        [ AC_MSG_ERROR(OpenSSL ENGINE support not found)]
                )


This generates code in configure lines 17887 onward:

  int
  main ()
  {
  
  int main(void){ENGINE_load_builtin_engines();ENGINE_register_all_complet        e();}
  
    ;
    return 0;
  }


which the Sun Studio compiler doesn't regard as valid C: it's
declaring a function 'main()' inside a function 'main()', and
terminates with an error.

gcc doesn't seem to mind compiling this code, but the Sun Studio
compiler isn't having it.


Fix:
configure.ac, line 1919

Lose the leading "int main(void){" and the trailing "}"
Comment 1 Andrew Benham 2006-09-29 00:07:59 AEST
Created attachment 1191 [details]
Fix for configure.ac
Comment 2 Darren Tucker 2006-09-29 20:20:58 AEST
I must admit that I didn't read what autoconf produced for this test, but I'm surprised that gcc accepted it.

Patch applied to both -HEAD and the 4.4 branch.  It will be next release.

Thanks.
Comment 3 Andrew Benham 2006-09-29 23:12:34 AEST
Well, apparently gcc supports 'nested functions', but the world is
divided over whether this is a good idea or not.
Whatever the individual view on this, it ain't standard C.
Comment 4 Damien Miller 2008-04-04 09:56:30 AEDT
Close resolved bugs after release.