Anybody understand cryptography?

This question has been nagging me for a while, and nothing I've read has helped me figure it out:

From what I understand, you use a public key to encrypt a message that can only be decrypted with the private key. But if the encryption software uses the public key to encrypt the message, and you can read the source code of the encryption software to see how it does that, why can't someone listening in just reverse the process using the public key, which they know because it's public?

I'm obviously missing something here, but I just can't figure out what it is.