It's All There In Black and White

Once again I'm hearing from people who say they won't read a blog or website if it's white text on a black background.

I'm guilty of both - I have it on my website and my blog. Does this mean I'm losing readers or potential readers?

I've tried putting an off-white background to my site, but my header is black/dark, and the results simply don't look good together.

I'm in a pickle. Any suggestions? Comments?