(It wouldn't let me just comment)
I would add, there are multiple cases where the apparent size of the skull and therefore brain do not correlate with intelligence, and this is across different species of the planet. If it were true, why are humans one of the apparently most intellectually evolved creatures? Consider whales. This cannot be explained by the aforementioned method.
If one were to look at brain size in the human population they appear to actually be decreasing in size over time, which could lead one to conjecture people aren't getting less intelligent but that the density of neurons is changing. The density is increasing. We can justify the lack of decreasing intelligence by the intelligence metrics that exist today, regardless of how flawed they may actually be, they show an average increase compared with the past.
So, in the human case, if you were take density into consideration then this altogether and at once dismisses the idea you speak of as being patently false.