VMWARE GSX doesn exist anymore. It is replaced by VMWARE Server which is free.

VMWARE server is a free virtualization software that run on a Windows Server platform. Good for testing and smaller environments

VMWARE ESX Is the Hypervisor from VMWARE.

VMWARE ESX server can run up to 128 VM’s at the same time. Your main limits are determined by how much RAM and CPU Power you have.

It has its own OS, so can not be installed upon Windows. But must be installed on the server itself. It uses it own file system: VMFS.
Has really nice features like Vmotion, HA and resource groups.

ESX is a server version and GSX is a client version, ESX is a linux kernal based os, GSX is a program which runs on windows, or even ESX