Short answer
Long-term memory formation (consolidation) takes weeks, or even years.
Background
Unfortunately my observations haven't been published yet, but an interesting thing I see with listening tests in noise is that people are systematically performing better when they come in a week later for a re-test.
Now I'll add some background information about types of memory as a general answer (Fig. 1).
The most relevant type here is long-term memory. There are a number of sub-types but one is most relevant for your question:
is defined as the memory system in charge of the encoding, storage,
and retrieval of the procedures (rather than episodes) that underlie
motor, visuospatial, or cognitive skills
It includes such arbitrary matter of understanding the test (listen and repeat sentences / play a game) and other when subjects perform a certain task for the first time.
Now an important part of long term memory is consolidation (Miller, 2008), which is
Memory consolidation is defined as a time-dependent process by which recent learned experiences are transformed into long-term memory, presumably by structural and chemical changes in the nervous system (e.g., the strengthening of synaptic connections between neurons).
Consolidation stabilizes memories after initial acquisition. Synaptic consolidation is the same as long-term potentiation and occurs within the first few hours after learning. Systems consolidation is the process where hippocampus-dependent memories become independent over a period of weeks to years.
After a while of playing and being past your peak, you may simply get tired, bored, or loose focus for other reasons, which reduces performance.
References
- Miller, Learning and Memory: A Comprehensive Reference
(2008); 1: 53–3
- Pitel et al., Handbook of Clinical Neurology (2014); 125: 211-25

Fig. 1. Types of memory. source: The Human Memory