From the retina, visual signals next travel through the lateral geniculate nucleus of the thalamus. This nucleus is quite centrally located, about as much in the middle of the brain as possible, and much much more ancient (evolutionarily, and developmentally) than the folded outer part of the brain that is highly recognizable in humans.
This entire folded brain region including frontal/parietal/temporal/occipital lobes is called the neocortex, and it is quite recent in evolution, only appearing in mammals.
In other vertebrates, the next step in visual processing is the optic tectum, also found in the center of the brain. In mammals we typically call this the superior colliculus, and it still functions in eye movements and visual target localization.
The position of visual cortex in humans is determined simply by the part of neocortex that gets direct inputs from the visual part of thalamus, so it is likely that it inherits somewhat from the organization of the thalamus, where you can notice that the LGN is located towards the back part of the thalamus, but note that LGN isn't simply the structure furthest back (MGN, the auditory thalamus, is further back, for example). Otherwise, there is no need or reason for the visual cortex to be in any particular part of the brain, only that it needs some space for itself apart from other regions. Input isn't coming directly from the retina in the eyes, so there is no improvement to path length to have the visual cortex anywhere else.
Vision isn't even forward-focusing in many mammals, this is a trait that you really only see in carnivores and primates that use binocular vision to make an improved 3D map of the world in front. Prey animals like rodents and many herbivores have vision that is more concerned with what is above or on any side rather than what is in front.
See also, to some extent, https://biology.stackexchange.com/questions/35532/why-do-some-bad-traits-evolve-and-good-ones-dont