When I build openssh-3.5p1 on a 32bit worksation running irix 6.5.18 sshd will crash with the error message "fatal: mm_malloc: size too big" on each login attempt. When I build it on a 64bit orign200 with the same OS version it works fine. The problem seems to be the constant ULONG_MAX which is assigned to SIZE_T_MAX in defines.h. I played around with the code and it seems like ULONG_MAX evaluates to -1 even though it is defined as 4294967295U (by the way: what does the U mean?) in /usr/include/limits.h. I worked around this problem by replacing #define SIZE_T_MAX ULONG_MAX with #define SIZE_T_MAX 4294967295 in defines.h before running make.
Sorry, typo: The irix version is 6.5.17 not 6.5.18
Same Problem on Solaris 8 Build 32-Bit, gcc 2.95-3 It seems that line 170 in monitor_mm.c is the problem: if (size > SIZE_T_MAX - MM_MINSIZE + 1) fatal("mm_malloc: size too big"); size seems to be a different type (uint32) in defines.h: --- #ifndef SIZE_T_MAX #define SIZE_T_MAX ULONG_MAX #endif /* SIZE_T_MAX */ #ifndef HAVE_SIZE_T typedef unsigned int size_t; # define HAVE_SIZE_T #endif /* HAVE_SIZE_T */ --- the line #define SIZE_T_MAX ULONG_MAX should be changed, I tried # define SIZE_T_MAX ((2 << (8 * sizeof(size_t)) - 1) but got an error (also out of bounds :-( The definition of SIZE_T_MAX has to be changed to be in the range of the type of size From the build log: --- gcc -O3 -pipe -mcpu=ultrasparc -m32 -I. -I. -I/usr/local/include -I/usr/local/i nclude -DSSHDIR=\"/etc/ssh\" -D_PATH_SSH_PROGRAM=\"/usr/local/bin/ssh\" -D_PATH_ SSH_ASKPASS_DEFAULT=\"/usr/local/lib/ssh/ssh-askpass\" -D_PATH_SFTP_SERVER=\"/us r/local/lib/ssh/sftp-server\" -D_PATH_SSH_KEY_SIGN=\"/usr/local/lib/ssh/ssh-keys ign\" -D_PATH_SSH_PIDDIR=\"/var/run\" -D_PATH_PRIVSEP_CHROOT_DIR=\"/var/empty\" -DSSH_RAND_HELPER=\"/usr/local/lib/ssh/ssh-rand-helper\" -DHAVE_CONFIG_H -c monitor_mm.c monitor_mm.c: In function `mm_malloc': monitor_mm.c:170: warning: integer overflow in expression monitor_mm.c:170: warning: comparison is always true due to limited range of data type --- -- Guido
I had to reformat Guido's post to read it: "Same Problem on Solaris 8 Build 32-Bit, gcc 2.95-3 It seems that line 170 in monitor_mm.c is the problem: if (size > SIZE_T_MAX - MM_MINSIZE + 1) fatal("mm_malloc: size too big"); size seems to be a different type (uint32) in defines.h: #ifndef SIZE_T_MAX # define SIZE_T_MAX ULONG_MAX #endif /* SIZE_T_MAX */ #ifndef HAVE_SIZE_T typedef unsigned int size_t; # define HAVE_SIZE_T #endif /* HAVE_SIZE_T */ the line #define SIZE_T_MAX ULONG_MAX should be changed, I tried # define SIZE_T_MAX ((2 << (8 * sizeof(size_t)) - 1) but got an error (also out of bounds :-( The definition of SIZE_T_MAX has to be changed to be in the range of the type of size From the build log: gcc -O3 -pipe -mcpu=ultrasparc -m32 -I. -I. -I/usr/local/include -I/usr/local/include -DSSHDIR=\"/etc/ssh\" -D_PATH_SSH_PROGRAM=\"/usr/local/bin/ssh\" -D_PATH_ SSH_ASKPASS_DEFAULT=\"/usr/local/lib/ssh/ssh-askpass\" -D_PATH_SFTP_SERVER=\"/us r/local/lib/ssh/sftp-server\" -D_PATH_SSH_KEY_SIGN=\"/usr/local/lib/ssh/ssh-keysign\" -D_PATH_SSH_PIDDIR=\"/var/run\" -D_PATH_PRIVSEP_CHROOT_DIR=\"/var/empty\" -DSSH_RAND_HELPER=\"/usr/local/lib/ssh/ssh-rand-helper\" -DHAVE_CONFIG_H -c monitor_mm.c monitor_mm.c: In function `mm_malloc': monitor_mm.c:170: warning: integer overflow in expression monitor_mm.c:170: warning: comparison is always true due to limited range of data type"
Created attachment 380 [details] Set SIZE_T_MAX to UINT_MAX if we we define size_t ourselves. defines.h is obviously wrong for the case where neither SIZE_T_MAX or size_t are defined; size_t ends up as unsigned int, while SIZE_T_MAX ends up as ULONG max. That works if sizeof(unsigned int) == sizeof(unsigned long). Any objections to this patch?
BTW, the "U" means the constant should be evaluated as an unsigned.
Created attachment 381 [details] Set SIZE_T_MAX to UINT_MAX if we we define size_t ourselves. Sorry, mixed patch.
Patch #381 was just committed, so I think this is now fixed. Please re-open if not.
Mass change of RESOLVED bugs to CLOSED