A couple years ago my boss asked me "What religion are you?" I said I was AGNOSTIC out of a general fear of using the A word in some situations. His answer caught me by surprise "An agnostic is just an ATHEIST who's afraid to commit." I was a little suprised to learn he was godless. I eventually learned that about half the engineers here are non-believers. The other half are uber religous. And still, I meet people that have never met an atheist (or so they think) Go figure.
Even though I do consider myself Atheist I say 99%. I hold out that 1% for proof lol. Not because I am afraid to commit but I don't want to be one that still denies if there is concrete proof. But I too find myself answering like you did and I find it sad. It's like we have to hide from the persecution of religious people for not believing in their imaginary friend in the sky. As I get older it's easier for me to say I don't do religion and let them read between the lines. And ya, I also think most people have actually met an Atheist.