Kim Peek (the real-life inspiration for Raymond Babbitt in Rain Man) had a spectacular case of savantism. He also had disabilities, like missing his corpus callosum and anterior commissure, leaving the hemispheres of his brain abnormally connected. I can't say they were less connected than in other people, since apparently unusual connections developed between his hemispheres, and if this netted him a higher or lower interhemispheric connectivity, I don't know. I say apparently, because this Wikipedia article mentions it, but the sources cited do not.
Here's my non-expert suspicion on the (at least primary) source of Peek's savantism: the lack or reduction of contrahemispheric inhibition, caused by the agenesis of the corpus callosum and anterior commissure, similar to how the Sprague effect works, where destruction of the right colliculus, or severing of the connection between the colliculi, abolishes the inhibition of the right colliculus upon the left colliculus (Kapur et al., 1996, p. 2). The removal of this inhibition allows for certain cognitive abilities to flourish, but it may have diminished other cognitive abilities (or maybe his disabilities were caused by other implications of his under-/maldevelopment). This explanation of mine may be completely false though.
Now, Peek's case is almost the opposite of other cases, where it is right hemispheric compensation for left hemispheric lesioning that causes savantism (Treffert, 2009, section h). For this to happen though, there must be a strong connection between the hemispheres to begin with, as the higher connectivity allows for a higher ability to redirect node strength to the right hemisphere in response to damage in the left hemisphere (Bartolomeo & Thiebaut de Schotten, 2016 p. 11)
The fact that the cause of savantism is in some cases the high or at least sufficient interhemispheric connectivity of the two hemisphere that causes enhanced cognitive function after unilateral lesioning, whereas in other cases it is the lack or reduction of contralateral inhibition (positively correlated with lower interhemispheric connectivity), tells me there's a great diversity in mechanisms leading to savantism, perhaps causing the great diversity of ability between different savant cases. The lack of a single theory then makes me wonder; are there criteria of cerebral affectation that must be met for savantism to occur? Are there certain mechanisms that underpin all savant cases?
This article makes it seems like all there is to savantism is injury to the left anterior temporal lobe (LATL), which reduces the LATL's suppression of the minutia of stimulus in favor of the "bigger picture" that's only concerned with the stimulus' implications on survival and/or social interaction (depending on which theory one is following). Given how this article is the most recently written of my citations here, perhaps this article shows the conclusion of what causes savantism, the other articles having been written in a time of uncertainty?