At its heart, sampling is basically taking a short audio clip and "playing" it as a musical instrument. Any audio source can be used as a sample.
As mentioned drum beats (for example taken from jazz/funk/old rock drum solos) are sampled in many current styles of music from rap/hip-hop to pop to many electronic genres (house, breakbeat, drum and bass).
Another source for sampling are other synths, and sometimes just pure waveforms (sine, sawtooth, pulse) which are then looped and loaded in a sampler. In this sence they're treated just like oscillators are used on synths, which then you use the filters and other synthesis parameters provided in sampler to further shape your sounds.
Samplers are also used to playback samples of acoustic instruments such as piano, guitar, orchestral instruments, etc. In these cases ideally you sample (record) each note of the instrument and multiple velocities (loudness), which are then mapped at the corresponding keys accross the keyboard. You can then play these instruments and if you're careful with your performances you can make these virtually indistinguishable from the "real" instruments.
The rest is up to your imagination... you can take nature recordings, sample movies, your farts, etc... load them into a sampler, and either play them as if they were musical sounds or mangle them further.
Samplers are very, very powerful tools.
Currently, pretty much all samplers have given way to their software equivalents. I'm not going to go into an argument whether they sound "good" or "bad" compaired to their hardware counterparts of the past, however, if you're interested in sampling, and you have a capable DAW/sequencer running, there are several to choose from.
The top of the list are:
Native Instruments' Kontakt
MOTU Mach 5
Steinberg Halion
Apple/eMagic EXS24 (although this only runs inside Logic)
E-mu EmulatorX (which requires one of their audio cards to run)
I'm sure there are many more, and there are some other specialized software, but that should cover for starters.
If you decide you want to go the hardware route, then you'll need to look for second hand AKAI or E-Mu samplers. They each have their own "sound", and if you require stuff like filters, you can't beat E-Mu.