Short answer
Contrast is hardwired in the visual system and can be explained by retinal and brain connectivities without the need for adaptive processes. My answer pertains to adaptation at the neurophysiological level. In other words, short-term neural adaptation in the retina or the visual cortex are not necessary components for color-contrast coding in the visual system.
Background
According to the model of Hering, color contrast in trichromatic species (such as most humans) is basically established between three sets of opponent systems: yellow-blue, red-green and the achromatic channel (dark-bright), and depicted in Fig. 1. The opponent system is established in the brain by cells in the visual cortex responding to both colors of a pair, but one color (e.g. blue) excites and the other (yellow) inhibits the cell, or vice versa. These systems are basically established by funneling the color information of opponent colors from the primary sensory neurons (the photoreceptive rods and cones) to color sensitive cells in the visual cortex (Gouras, 2009). In effect, we cannot perceive a yellowish blue, or a reddish green, because these colors are opponent colors in a pair.
Fig. 1. Hering model of color opponency. source: Mark Green
Because of this, the primary color-pairs yellow-blue and red-green yield excellent contrast, because in the brain the opponent response of cells will cause the edge to be sharply defined. Likewise, the achromatic channel yields high-contrast acuity. Mixed colors will impinge on this system not in an ON/OFF way, but in a gradient. For example, in your example, the blue chalk will be visualized through the yellow-blue opponent channel. On a black background, color contrast will be established against the achromatic channel. The highest possible contrast would be delivered with the blue chalk on a yellow background such that the blue-yellow opponent system could be used (e.g. blue chalk on a yellow 'white' board :-). However, blue chalk on a blackboard had to be contrasted using the yellow-blue against the achromatic channel, which are phsyiologically not opponent systems and therefore yield less sharp contrast.
Another reason is foveal tritanopia: the fovea has the highest resolution for perceiving fine detail. There are no short wave cones (blue cones) in the very center, the area of maximum resolution (Williams et al., 1981), so presumably it is impossible to see blue with the area of the retina with the highest contrast acuity.
Note that while the aforementioned Hering model does not depend on adaptation, many illusions do depend on adaptive processes in the color system, most notably after-images.
References
- Gouras, Color Vision. In: Webvision. The Organization of the Retina and Visual System (2009)
- Williams et al., Vis Res (1981) 21(9): 1341–56