mm/huge_memory.c in the Linux kernel before 2.6.38-rc5 does not prevent creation of a transparent huge page (THP) during the existence of a temporary stack for an exec system call, which allows local users to cause a denial of service (memory consumption) or possibly have unspecified other impact via a crafted application.
http://openwall.com/lists/oss-security/2011/02/17/3
http://openwall.com/lists/oss-security/2011/02/17/6
http://www.kernel.org/pub/linux/kernel/v2.6/testing/ChangeLog-2.6.38-rc5
http://www.securityfocus.com/bid/46442