Your question is about what is called validity: the idea that a measure actually measures the thing it purports to measure.
Your explanation seems limited to a particular type of validity called face validity: that something appears to measure the thing it purports to measure, on its face/"just by looking at it".
Importantly, face validity doesn't necessarily mean a test is valid in other senses - measures might seem like they work but actually don't. Similarly, a lack of face validity doesn't mean a test does not have a different form of validity.
Measuring intelligence is very difficult, perhaps impossible given there is no need for agreement on what "intelligence" measures. Generally, though, the validity of intelligence measures is assessed by how well they correlate with other intelligence measures, and how well they correlate with longer-term outcomes: academic achievement, high-wage employment, etc. The value of a test like Raven's Matrices is that it's fairly quick and easy to administer; it's also non-verbal, so it can be used by people with different language backgrounds, and doesn't require a lot of specific domain knowledge so is appropriate across age and background. The downside of any one intelligence test is that they typically measure only a slice of what could be considered intelligence, so results should be interpreted carefully, and consideration should be given to whether the test is valid for the specific purpose it is used, rather than valid more generally.
There are lots of papers on the validity of Raven's Matrices in various contexts.
Additionally, for your specific example:
I have a bag filled with some objects stickered with numbers from 1 to 10. These objects may or may not have anything to do with each other. If I were to, say, give you the objects corresponding to label 1-3
This is not an item in Raven's Matrices, so it should not be used to critique that test, even on face validity grounds.