The first important thing to note is that stock SunOS 4.1.4 does not appear to ship with libXtst.so or an X server that supports the XTEST extension. In that case x11vnc can only provide view-only access to the display (if the display has the DEC-XTRAP extension then it may be possible to use that see the -xtrap option).

But very often the MIT X11R5 release is built and used on this release of SunOS. In that case the MIT X server supports XTEST (and must be enabled and running), and libXtst.so will be available (perhaps in /usr/X11R5 or /usr/local). Be sure to set CPPFLAGS and LDFLAGS to find the the X11R5 libraries and header files. See the general build notes for how to pass these locations to the configure script.

As with Solaris 2.5.1 and earlier some lines have to be added the autogenerated rfb/rfbconfig.h file in the source tree, and just before the last #endif at the bottom of that file. We found this worked for SunOS 4.1.4:

#include <sys/types.h>
/* typedef unsigned int in_addr_t;   only x11vnc 0.7.1 and earlier */
struct timeval _tmp_usleep_tv;
#define usleep(x) \
    _tmp_usleep_tv.tv_sec  = (x) / 1000000; \
    _tmp_usleep_tv.tv_usec = (x) % 1000000; \
    select(0, NULL, NULL, NULL, &_tmp_usleep_tv);
#undef LIBVNCSERVER_HAVE_LIBPTHREAD
int gethostname(char *name, int namelen);
long random();
int srandom(unsigned int seed);
#define SHUT_RDWR 2
#define snprintf(a, n, args...) sprintf((a), ## args)   /* implies gcc */
typedef int ssize_t;
#define memmove(dst,src,len) bcopy((char *)(src),(char *)(dst),(int)(len))
extern char *sys_errlist[];
extern int sys_nerr;
#define strerror(n) \
    (((n) >= 0 && (n) < sys_nerr) ? sys_errlist[n] : "unknown error")
/*
 * X11R5 libXtst does not have XTestGrabControl.   The x11vnc configure
 * will try to use DEC-XTRAP extension for grab control.  If that doesn't
 * work uncomment this line to get x11vnc to build.

#define XTestGrabControl(a, b) 1

 * NOTE that if any app (e.g. wm) grabs the pointer or keyboard, that can
 * freeze x11vnc.  Try to disable grabs for your window manager, etc...
 * (one could even consider using xwit(1) to manipulate windows...)
 */

We also found for some X11R5 installations we needed to add "-I/usr/openwin/include/X11" to CPPFLAGS and "-L/usr/openwin/lib -lX11 -lXt" to get the DEC-XTRAP stuff to compile. Why libXt comes in is not clear...

A strange /usr/bin/sh bug appeared when running the x11vnc configure script:

# ./configure
./configure: /tmp/sh104535: No such file or directory
./configure: /tmp/sh104536: No such file or directory
./configure: /tmp/sh104537: No such file or directory
./configure: /tmp/sh104538: No such file or directory
./configure: /tmp/sh104539: No such file or directory
./configure: /tmp/sh1045310: No such file or directory
./configure: /tmp/sh1045311: No such file or directory
./configure: /tmp/sh1045312: No such file or directory
configure: loading cache /dev/null
checking for a BSD-compatible install... /bin/install -c
checking whether build environment is sane... yes
./configure: /tmp/sh1045313: No such file or directory
sed: conftest.sed: No such file or directory
rm: conftest.sed: No such file or directory
checking for gawk... no
checking for mawk... no
checking for nawk... nawk
checking whether make sets $(MAKE)... ./configure: /tmp/sh1045314: No such file or directory no
./configure: /tmp/sh1045315: No such file or directory
./configure: /tmp/sh1045316: No such file or directory
checking for gcc... no
checking for cc... cc
./configure: /tmp/sh1045317: No such file or directory
./configure: /tmp/sh1045318: No such file or directory
checking for C compiler default output... configure: error: C compiler cannot create executables
See `config.log' for more details.

# env SHELL=/usr/local/bin/bash /usr/local/bin/bash ./configure

BTW, if your SunOS 4.x machine is a slow sun4m (or earlier), the x11vnc performance will likely be unacceptable (long pauses if much of the screen changes). Performance will be better for 8bpp Pseudocolor on a smaller display (say 1024x768) compared to 32bpp TrueColor at 1280x1024.