Here is a YouTube video summarizing his experiment.
Over a number of years, the ethologist John B. Calhoun conducted over-population experiments on Norway rats (in 1958–1962) and mice (in 1968–1972). His studies describe the collapse in behavior which resulted from overcrowding.
● Calhoun, J.B. (1962). Population density and social pathology. Scientific American, 206, 139—48.