I’m not sure that there is an answer to this question in the literature, and I doubt that we’ve ever studied cases of “running out of memory”. But it’s an interesting question, so I’ll venture to answer it. However, my answer, at best, will be just an assumption based on current knowledge and based on similar situations in other cognitive functions. Yet, I hope that it will give some ideas regarding this question. Or, better yet, other people will provide more comments and ideas that improve this answer.
We may never have seen cases of “running out of memory”, but we all certainly have encountered situations that we run out of our abilities to do something, such as to break our own sport records, to learn to speak a foreign language, and to memorize as many digits as we can in a single glance. What does the brain do when it encounters situations that it runs out of its abilities? The answer is it adapts itself by the process called neuroplasticity (ref 1, ref 2). Although, generally, our abilities will improve to some degree, this neuroplastic adaptation takes time and does not guarantee that it will be successful (e.g., we may never be able to break the Olympic running records ourselves or to remember 1,000 digits in a single glance) because we all have biological limits. What does the brain do when it reaches those limits? From observations, I think it just accepts the limits and does nothing more.
From this, it is logical to think that if anyone really runs out of memory, the first thing that the brain does will probably be not rewriting the old memory but modifying its memory synapses because it has the ability to form new memory form modifying its memory synapses (ref 3, ref 4, ref 5). But after some time, it may reach its ability to further modify its synapses for new memory, what happens then? Does it rewrite the old memory? I have no references to answer this question, but I doubt that it does. To rewrite the old memory means to destroy the previous structure of synapses that hold the old memory and build the new structure of synapses to hold the new memory. The brain will end up having the same amount of memory but wasting its resource to do that. Is it worth it? Also, there are technical issues such as which old memory to be erased, how to locate the scattered synapses that hold the to-be-erased memory, and how to dismantle those synapses. Perhaps it’s best to just accept its limits and do nothing more as in the other situations above.
References
- Adult Neuroplasticity: More Than 40 Years of Research
“Today it is generally accepted that the adult brain is far from being fixed. A number of factors such as stress, adrenal and gonadal hormones, neurotransmitters, growth factors, certain drugs, environmental stimulation, learning, and aging change neuronal structures and functions. The processes that these factors may induce are morphological alterations in brain areas, changes in neuron morphology, network alterations including changes in neuronal connectivity, the generation of new neurons (neurogenesis), and neurobiochemical changes. Here we review several aspects of neuroplasticity and discuss the functional implications of the neuroplastic capacities of the adult and differentiated brain with reference to the history of their discovery.”
- Changes in plasticity across the lifespan: Cause of disease and target for intervention
“We conceptualize brain plasticity as an intrinsic property of the nervous system enabling rapid adaptation in response to changes in an organism's internal and external environment. In prenatal and early postnatal development, plasticity allows for the formation of organized nervous system circuitry and the establishment of functional networks. As the individual is exposed to various sensory stimuli in the environment, brain plasticity allows for functional and structural adaptation and underlies learning and memory. We argue that the mechanisms of plasticity change over the lifespan with different slopes of change in different individuals.”
- Synapses and Memory Storage
“We now understand in considerable molecular detail the mechanisms underlying long-term synaptic plasticity and the importance that such plastic changes play in memory storage, across a broad range of species and forms of memory. One surprising finding is the remarkable degree of conservation of memory mechanisms in different brain regions within a species and across species widely separated by evolution. However, although it is now clear that long-term synaptic plasticity is a key step in memory storage, it is important to note that a simple enhancement in the efficacy of a synapse is not sufficient to store a complex memory. Rather, changes in synaptic function must occur within the context of an ensemble of neurons to produce a specific alteration in information flow through a neural circuit.”
The Corticohippocampal Circuit, Synaptic Plasticity, and Memory
“Synaptic plasticity serves as a cellular substrate for information storage in the central nervous system. The entorhinal cortex (EC) and hippocampus are interconnected brain areas supporting basic cognitive functions important for the formation and retrieval of declarative memories. Here, we discuss how information flow in the EC–hippocampal loop is organized through circuit design. We highlight recently identified corticohippocampal and intrahippocampal connections and how these long-range and local microcircuits contribute to learning. This review also describes various forms of activity-dependent mechanisms that change the strength of corticohippocampal synaptic transmission. A key point to emerge from these studies is that patterned activity and interaction of coincident inputs gives rise to associational plasticity and long-term regulation of information flow. Finally, we offer insights about how learning-related synaptic plasticity within the corticohippocampal circuit during sensory experiences may enable adaptive behaviors for encoding spatial, episodic, social, and contextual memories.”
Learning to learn – intrinsic plasticity as a metaplasticity mechanism for memory formation
“Use it or lose it” is a popular adage often associated with use-dependent enhancement of cognitive abilities. Much research has focused on understanding exactly how the brain changes as a function of experience. Such experience-dependent plasticity involves both structural and functional alterations that contribute to adaptive behaviors, such as learning and memory, as well as maladaptive behaviors, including anxiety disorders, phobias, and posttraumatic stress disorder.”