Firstly make sure you are on ESXi 5.0 update1. Create a virtual machine and set the OS to "Microsoft Windows 8 (64-bit)". At the end of creating the VM, us the check box"open setting dialog"
In the VM's settings:
- Change the video card to Auto detect settings and check the "Enable 3D support"
- go to the Options->Boot Options and change the firmware to EFI
I didn't modify the settings on the VM the first time I tried to install Windows 8 and ended up having to reinstall after making the above changes.