Personally and philosophically, I reject that anything can be truly known, only that it can thought to be known. I'm very harsh on what can and can not exist. For example I don't hold that "I think therefore I am" is necessarily true.
But hey that's weird stuff for you. In a framework where things can be "known", that is relatively close approximations can be mentally constructed, and let's face it, maths (logical and otherwise) and the overarching scientific method is the best way of doing that... God appears to be a superfluous concept. There is no thing that specifically requires a theistic god that cannot be explained by simpler means. By process of induction, it is phenomenally unlikely that something like a deistic god could have existed (if all these things can be explained by natural processes according to arbitrary laws of nature, is it more likely that everything is that way, or that somewhere along the way, some thing requred a god with it's own unknowable rules?). Pandeistic, hard to determine (even more unfalsifiable), however there's no point in worshipping a pandiestic god, so we're safe with the temporary conclusion that it doesn't exist based on lack of evidence.
All of these start with the assumption that they are false. This is partly because philosophically, it's the only safe assumption to make, and it's the safer starting point for logic, and also because it's the mandated starting point of the scientific process, for reasons you have so nicely stated (and mostly because of the logical and philosophical reasons).
Taking us all the way back to the beginning, the button has all the evidence required to prove the existence or nonexistence of a or any god(s). What I can't get is why you would actively reject all reason to believe whatever you want to believe, rather than what is demonstrably true, when it's clear we all agree on what the foundations of "truth" are.
Do we? I wouldn't be so sure...
To summarize: my set of principles are my own choice (just as yours are yours), by virtue of my intrinsic liberty. I don't necessarily want to play "safe" all the time (why do people climb Everest, anyway?) because, as I said, we are all gamblers, and gamblers take (more or less calculated) risks, with the expectancy of a reward of some kind. No risk, no reward. What I argued was that the only way you can legitimately attack my philosophical stance, and whatever I hold as "the truth", is by evidencing some kind of inconsistency or flaw in what I maintain, e.g. by falsifying one of my statements (presumably derived from my principles, or from their consequences). Unless (or until) that happens, we can both make our own choices and hold our own separate "truths", that may not entirely coincide, nor totally diverge either.
If you "reject that anything can be truly known", that's quite a different statement from "nothing exists" (isn't it?), and something that I can understand and subscribe to. That's why science itself is not static, but ever evolving. A theory is just an approximation of reality (like you said), and any scientific statement is susceptible of being falsified (or rephrased in the context of a larger theory) at some point in the future. But something of it will survive. We have countless examples of that. E.g. the newtonian mechanics was replaced by the quantum mechanics, or by the relativity theory, which doesn't mean it was completely wrong. The classic mechanics can still describe the reality within certain restrictive conditions, and you could still send a man to the moon (but not likely to another galaxy ) using newtonian equations. Similarly, "the sky is blue" is a scientific statement given the right observability conditions. It can be restated as: "if we're on planet Earth and look above on a sunny day, we're going to see the color blue all above the horizon; we call it the sky".
To come to the point. What I hold as the truth is a set of (implicit and explicit) principles and personal choices, along with their derived consequences, out of which no statement that I may make has yet been falsified (to my knowledge). But that's just on the surface; it's the part that I can legitimately show to others. Beneath that, my real (personal) belief is that life, or the universe / existence, or us, are more than, and cannot be reduced to, a bunch of schemes / theories, no matter how comprehensive. The "truth" (i.e. anyone's truth) is simultaneously objective and subjective, and it is also not manifest (meaning that our ontological condition does not allow us to speak the truth just by opening our mouth). We have to undertake a great deal of effort and pain, in order to (begin to hope to) reach it. Which is the reason why theories and quests of the "truth" will never come to an end (and not only because of Godel's theorems), as long as we're humans. For the objective (quasi-deterministic) component of the truth, we have science, which is a great tool to pursue it. But as far as the subjective part (associated with our intrinsic liberty) is concerned, there's a little problem: it doesn't comply to the principle of observability (in the reproducible form of a scientific experiment), therefore it cannot in principle be the object of science. Which is why I'm convinced that science, or even logic, will never be the container of the truth as a whole; a part of it (sadly, the part that ultimately matters) will always escape them. Things like the sense of existence, moral values, art, ultimate questions about the meaning of life and death, or "that which matters most" of Plotinus, will never just vanish from our preoccupations and quests, because they are a part of the "truth" (which, in its higher strata, is a synonym of "beauty" and "good") that makes us who we are. From this point on, we enter the realm of paradox and aporia. So if you ask me about my vision of the ultimate truth (a question equivalent to that of the meaning of existence / life), I will only say that it's the point where the subjective and objective components of the truth merge, like in the metaphor about "finding a door in a wall that has none".
You can, of course, simply disregard everything that I've just said in the last few sentences (where I put down my social mask for a moment) as irrelevant, crazy, inconsistent, or pure nonsense. That's your choice.