With the typical built-in microphone input on a PC / laptop, noise is usually part electronics noise and part power supply noise.
A cheap or poor quality microphone will only add to that.
If you turn the gain up, any of those sources will also be boosted.
You could try a cheap USB sound module (for both microphone & headphones) as they have somewhat better isolation from the "noisy" internal electronics in the computer/PSU.
Either of these or similar styles may be worth a go:
http://www.ebay.co.uk/itm/External-Sound-Card-USB-6-Channel-5-1-Audio-S-Pdif-PC-Netbook-Laptop-UKS-PDIF-/360585238790?hash=item53f48e1906:g:dFwAAOSwqu9U9JHm
http://www.ebay.co.uk/itm/External-Virtual-USB-3D-5-1-Channels-Stereo-Sound-Card-Audio-Adaptor-Converter/171898196952?_trksid=p2045573.c100505.m3226&_trkparms=aid%3D555014%26algo%3DPL.DEFAULT%26ao%3D1%26asc%3D38661%26meid%3Da83e0aa4930c4b5184f70af70f7d1bc7%26pid%3D100505%26rk%3D1%26rkt%3D1%26
(Ignore the "3D" / 5.1 junk on the second, it's a straightforward microphone + headphone interface. The rest is daft software gimmicks.)
Failing that, you need either a music-grade USB audio interface or a USB headset..
Something like this, with an appropriate quality microphone, will give superb results:
http://www.ebay.co.uk/itm/ROLAND-EDIROL-UA-25-USB-MIDI-AUDIO-INTERFACE-CAPTURE-24-BIT-/272471543842?hash=item3f7091d022:m:m1DVuKukPsyYuWYF-v8-TXg