The only words in Chinese that I know are "Ni How" (hello) and "she she" (thank you). And the only words I know in Japanese are "Domo arigato Mr. Roboto" (thank you very much Mr. Roboto). Yet I am able to tell the difference between when a person is speaking Chinese and when a person is speaking Japanese.
Japanese is not very tonal, is spoken very fast, and has relatively few phonemes. Consonants are often followed by a vowel. Chinese on the other hand is highly tonal and is spoken relatively slowly, as information is encoded extensively in tone. They are almost complete opposites. There are also various phonemes that occur more often in one language than the other, and emphasis is very rare in Japanese (e.g. it's "wa-ta-shi", never "wa-TA-shi"). It is also very regularly paced, with each syllable usually taking the same amount of time. Chinese varies the syllable lengths considerably.
Compare the following (romanized) Japanese
mou ichido itte morae masu ka
with the equivalent (romanized) Mandarin Chinese
qǐng zài shuō yíbiàn
It's clear that they are different languages, and that one is far more tonal than the other. These both mean the same things, asking the speaker to repeat what they had just said.