One interesting way to study transparency is by looking at cases where we perceive transparency even though there isn't any objective transparency.
Take this illusion from Adelson (2000) as an example.
In the both panels the diamonds have the same physical brightness (pixel value on the monitor), but our perception of the diamonds on the right is warped by the seemingly-transparent dark stripe that covers the diamonds on the top. Thus, even low-level image features like brightness are perceptually warped by the way our brain perceives the structure of the image.
There are a large number of papers on mathematical models of illusory transparency perception. A starting point might be Anderson (1997) and the references within. But, the general message from this research is that we sometimes perceive transparency as the result of imposing some kind of higher-order structure to make sense of the image.