The "social brain" concept likely originated with Robin Dunbar's Social Brain Hypothesis, proposed in 1998. However, Dunbar was not suggesting that the brain has dedicated "social modules". Rather, the hypothesis was that the evolutionary driving force behind intelligence (increased brain size and capacity) was social - most prominently the need to track relationships in communities of up to 150 people (henceforth known as Dunbar's number). However, I'll focus my answer on the question as asked.
Plasticity:
If the criteria for "inherently" social requires some "module" of the brain that has no value outside of a social context, then it is difficult to argue that the brain is "inherently" anything because the brain is so plastic (contrast this view with nativism).
For example, say we argue that the brain is "inherently" visual as there is a fairly distinct visual cortex dedicated to this function. However, in blind people and others, this area can be co-opted for auditory, tactile, and spatial processing, so there is nothing "inherently" visual about it. Perhaps we should focus on a "lower level" part of the visual processing system, such as the LGN, that really is only ever used for visual processing.
Language:
It is difficult (though not impossible) to imagine evolutionary value for language and speech production outside a social context, and this function does appear to have domain-specific modularity in the brain.
For example, Broca's area is considered the "motor speech cortex", and appears to be dedicated to speech production. But like most areas of the brain, the boundaries of this region are not well defined, and on closer inspection, neither is its function - although just about all functions attributed to Broca's area are social. While language processing has for a long time been associated with Broca's and Wernicke's areas on the left side of the brain, language processing can spread out as far as the right hemisphere, and some individuals with lesions in these regions can retain largely unaffected language capabilities.
Relating:
The FFA, in the fusiform gyrus, is another potentially social part of the brain, critical for face recognition - a lesion in this region leads to prosopagnosia (face blindness). This function is distinct from the system for recognizing inanimate objects, such that other visual agnosias may not affect facial recognition ability. This function is already developed in infancy, and is a strong candidate for domain-specificity. However, once again, this region of the brain may have other functions, and facial recognition can be subsumed (although less efficiently) by other regions.
Fusiform face area
Both autism and Asperger's provide another hint at the social nature of the human brain. Though their etiology is not well understood, it is considered biological, suggesting a disruption of some inherent social mechanism of the brain. However, since ASD affects other cognitive functions as well, it is again not clear that a distinctly social element of the brain is involved.
Learning:
Both social reward and social pain have been shown to largely recycle the non-social reward and punishment systems, suggesting that there may be nothing inherent in the brain to support learning by social motivation. However, as pointed out in the question, Theory-of-Mind capabilities may have some modular support in the brain, though no domain-specific mechanisms have been implicated.
Noam Chomsky famously argued that language acquisition in humans happens far too quickly to be accounted for by simple learning - that is, our brain must be somehow predisposed for language acquisition. How this happens is still not well understood, but the same logic is applicable to many other aspects of our social capabilities: We are simply too well honed at social functions such as language, relationships, modelling, and other social behaviours to be accounted for by learning / culture alone.
