Christians tend to think that if you're not religious, you're atheist.
Maybe this gives them hope for my soul.
Atheists, conversely, tend to believe that if you're not atheist you must be religious.
You can accept a simple label such as "agnostic" and it will make you sound intelligent and allow you to duck the issue.
The issue is that belief underlies it all.
It's like there's a strange tripswitch, a sequence of human thinking that skips to the one most-acceptable conclusion: if you're not one then you must be the other.
And, no matter what you believe or if you don't you need a label.