To learn about how a web server actually works, I'd suggest setting up your own VPS on Linode. They've got a great library of information to help someone get started: http://library.linode.com
If you're coming from a Windows background and don't have any Linux/Unix command line experience, though, this will be daunting.
I spent all of yesterday trying to upgrade ruby/rails on my linode for the first time. Kept running into errors and eventually decided to just start over with a clean Ubuntu install. That kind of demoralizing experience isn't a problem when starting with Heroku.
RubyGems and Linux (especially Debian / Ubuntu) are like oil and water.
Don't install Ruby libraries through apt-get, and also don't install them through RubyGems globally (i.e. never use "sudo" for installing gems).
Instead install gems for the local user only, with the help of RVM. Then to upgrade, if everything breaks, you can just delete your RVM directory and start from scratch again.
Of course, this brings with it a whole other can of worms, because then it's your responsibility for upgrading your libraries with the latest security-related fixes (which otherwise would have been updated with a simple "aptitude safe-upgrade", which can be a cronjob or something). But then you've got the same responsibility on Heroku.
