Baddeley's model suggests two buffers in the working memory: a phonological loop for acoustic, and a visuospatial sketchpad for visual information. The evidence is that auditory interference caused more performance decrease in auditory rather than visual tasks, and vice versa.
But I don't think that this is a very big evidence for the theory, that is because, instead of two separate buffers for different kinds of information, perhaps the information held in working memory is more susceptible to similar information, and there is only one storage unit in the working memory.
For example, it is possible that a verbal task in French can more easily be affected by a French rather than English interference. (well... maybe this was not a good example, because there are many external factors here)
Is there any study which supports this idea? I couldn't find any, probably because my choice of keywords was not good.