A concept has the property of depth, which is basically how much background understanding must be recruited to interpret it. The deeper the concept, the more understanding is required. Sheldon talking complex physics is relating many deep concepts.
Your mind tries to make sense of new information by associating it with your background understanding of reality. If you lack the background, it will have to work harder to make sense of it, and if it can't make sense of it, you will find yourself lost.
When new information is taken in and used in various ways (eg. muilling it over; using it in everyday life), redundant portions of it become incorporated into the unconscious mind to be used as a framework to further understanding (i.e. it becomes "second nature"). Unconscious information processing is much faster than conscious processing, so while you lose conscious access to the concept, you gain fast processing speed effortlessly.
When Sheldon speaks, he is accessing a deep unconscious network of understanding.
The way to increase speed of processing is to use related information repeatedly until your mind builds up the background understanding. Much of the more redundant processing will be relegated to the unconscious, freeing up your conscious mind to take in more information.
In other words, you lack the requisite foundation to understand Sheldon's complex physics in realtime, so are trying to consciously process everything said. Practice, practice, practice thinking about complex physics to build a foundational understanding that'll take the load off to speed up processing.