More on this...the digits 0-9 are just symbols for some common integer quantities. There's 10 symbols, which is sufficient for a base-10 positional number system to represent most numbers as a finite string of digits. This is only relevant in mathematics as a way to represent a specific number, however. Mathematics can prove, for example, that n digits in base b can represent b^n different quantities. It doesn't care how you represent b and n. Mathematics can prove that no finite decimal representation can represent 1/3. This doesn't prevent mathematics from handling the quantity 1/3.
People doing arithmetic generally handle base-10 representations of numbers, but they could do so in binary, balanced ternary, using Roman numerals or tally marks, etc. People doing mathematics handle numbers and other more complex concepts symbolically using clearly and unambiguously defined rules and rigorous logical reasoning, and often go quite far doing so without explicitly referring to any specific numbers.
You can't even begin the simplest mathematical proof or justify mathematical operations without logic, and logic so naturally fits in a mathematical framework that it can only be called a part of mathematics. You can't separate the two.
Now we're doing science, and science is based on making rigorously defined predictions and then testing them against reality by means of taking measurements. If your theory is "non numerical", how does it make quantitative predictions? If your theory does not yield quantitative predictions, how is it to be tested?



