Disclaimer:
~ Firstly I would like to say that there could be many factors as to why your brother is least attached to your parents, that may have nothing to do with the separation between 0-3 years. ~
For the purposes of keeping this succinct, I will discuss only the effects of bonding from 0-3 years of age.
Introduction:
This is an interesting question, as much research has been done on the effects of child abuse and neglect on attachment, in this case you are discussing a (I'm assuming) healthy child from within a loving family environment.
A number of factors would also affect the attachment of such as child, in terms of why he was separated from his parents for those years.
Overview of Development:
Firstly there is no denying that the first three years have significant affect on how brain development, including attachment styles.
I have added yellow boxes for highlighting.

Image courtesy of:
Rethinking the brain: new insights into early development - book by Rima Shore
Attachment 0-3 years:
Studies would suggest that the first three years are vital for forming attachments with caregivers. A child cannot form an attachment with someone they have not met.
A central component of attachment theory is the notion that caregivers
must be present and accessible in order for their children to become
attached to them. (1)
Separation:
It has been shown that any separation between mother and child within the first three years is detrimental to that relationship. This is assuming the mother and child already have a relationship.
His early work demonstrated that separations as brief as one week in
duration could negatively impact the quality of the relationship
between mother and child (Bowlby, 1969/1982). (1)
So the same theory would apply for other caregivers in the first three years of life, in this case the grand parents.
According to attachment theory, a secure attachment is derived from
the child’s appraisal of his/her mother’s (or other attachment
figure’s) availability. (1)
At this age, a child is not able to reason and can only process experience emotionally, so any parental absence is beyond comprehension of the child. The more damaging aspect of this, however, is the removal of the child at the age of three years from his/her primary caregivers (in this case the grand parents). I do not know what the situation was after your brother was returned to his parents, so it would be conjecture to discuss this in any detail.
As previously mentioned, the reason for the separation, where the
child stayed, and who provided care are all potentially informative
for helping to understand the effects of separation on parents and
children. A maternal separation is quite likely not as distressing to
an infant if he/she is left in the care of another attachment figure
to whom he/she is securely attached (1)
In conclusion:
It would seem that a child's complete separation from his/her parents until three years of age, would have an impact on the degree of attachment within those relationships. The people in the childs life in the first three years of life, would have closer attachments.
An observations, it is even shown in cases where children are left in abusive homes fare better with attachment styles than children who are removed and placed into foster care. Given this, the disruption to early relationships between child and caregiver is significant and has long term outcomes; in this case, the long term effect of such absence would damage the attachment to his parents.
References:
(1) Early Mother-Child Separation, Parenting, and Child Well-Being in Early Head Start Families
Kimberly Howard, et al PDF
doi:10.1080/14616734.2010.488119
Reversing the Real Brain Drain
Early Years Study PDF
Unicef, Early Childhood Development
The key to a full and productive life PDF