The human brain has a finite quantity of neurons (86 billion being the last estimate), which can each be in a finite amount of states (even accounting for current models of quantum physics and quantum computation, and hypothesis that those do play a non negligible role in the functioning of the brain in general and memory in particular).
The research area of Information Theory has dealt with "the quantification, storage, and communication of information" since the 1920s. It has yield many robust applications (e.g. data compression) which you use every single day, including when downloading and reading this stack-exchange post. A key concept in information theory is information entropy, the average level of "information", "surprise", or "uncertainty" in the output of a system (such as, for instance, a memory); and a key property is that albeit a finite system (say, of $n$ components of $b$ states each) can hold at most an exponential amount of information (here, $b^n$ bits of information), this very large amount is nevertheless finite.
I can only guess that the people you cite as describing the amount of information that a brain can hold as infinite are at once
- confusing "exponential" (which can get to very large quantities very "fast", including numbers larger than the estimated number of particles in the universe) with "infinite"; and
- confusing an upper bound on the amount of information that a system (here the brain) can hold with an actual measurement of its holding capacity.
Hope it helps!