It is neither:
Aphantasia is the inability to voluntarily create mental images in
one's mind.
How this relates to visual memory is still under investigation, as is most everything about aphantasia, a condition first described only in 2015.
The original paper coining the term by Zeman, Dewar, & Della Sala (2015) did speculate about the connection:
The descriptions given by our participants suggest that in some visual
memory is preserved even if visual imagery is absent, while others may
rely entirely on non-visual representations in memory tasks ...
A few studies since (eg, Keogh, Wickena, & Pearson, 2021; Bainbridge et al, 2021; Dawes et al, 2020; Pounder et al, 2021; Zeman et al, 2020; Milton et al, 2021) indicate that visual working memory and visual spatial memory are largely unaffected by aphantasia, while autobiographical and object-specific memory may be impaired.
Researchers currently speculate that aphantasics encode visual memories using strategies that differ from those with typical mental imagery, resulting in recall that sometimes matches and other times underperforms the general population. Accordingly, a recent study by Monzel, Vetterlein, & Reuter (2021) demonstrated that memory tasks where substitute strategies are less effective, are much more challenging for aphantasics.