The HandleRFBServerMessage function in libvncclient/rfbproto.c in LibVNCServer 0.9.9 and earlier does not check certain malloc return values, which allows remote VNC servers to cause a denial of service (application crash) or possibly execute arbitrary code by specifying a large screen size in a (1) FramebufferUpdate, (2) ResizeFrameBuffer, or (3) PalmVNCReSizeFrameBuffer message.
http://www.oracle.com/technetwork/topics/security/bulletinoct2015-2511968.html
http://www.openwall.com/lists/oss-security/2014/09/25/11
http://www.ocert.org/advisories/ocert-2014-007.html
http://www.debian.org/security/2014/dsa-3081
http://ubuntu.com/usn/usn-2365-1
http://secunia.com/advisories/61682