The church claims that it was there from the beginning and that it was not forgetful about many important matters. I agree with this premise.The problem comes when modern churches and Christians claim more than what was there in the beginning. Because certain ideas were present does not mean they were universally embraced by all ancient Christians and teachers. Such is the case regarding Mary and the holy family.
Earliest tradition suggests that Mary remained a virgin because of the prophecy of Ezekiel 44:2. The prophet said, "This gate shall be shut, it shall not be opened, and no man shall enter in by it; because the Lord, the God of Israel, has entered in by it. Therefore, it shall be shut." Some evangelicals will balk at this reading of the Old Testament but again the early Protestant Reformers read the text in this way.
Joseph has generally been considered as older than Mary. This is suggested by the fact that he had apparently died by the time of the crucifixion. He is in the infant story and then disappears sometime after Jesus was twelve years old. James was very likely an older brother, not a younger one. The time-line of his conversion suggests that he came late to believe in Jesus. Some have argued that younger siblings , including those five mentioned in Mark, would have attached themselves to an older brother of such charisma as Jesus. This would likely have happened from childhood on. Just think about the episode in the temple and you see the implications here. Only James remains in the record of the early church. Tradition says he was the author of the New Testament letter by this name.
It is hard for me to imagine that Jesus had younger brothers and sisters. If Mary and Joseph had other children (naturally) then the church would have very likely kept a record of them in some way. It makes sense given the Jewish culture and the role of family. Considerable scholarly discussion has taken place around this question but the most natural understanding would be that they never had children together.
Catholic teaching says the womb that brought the one who is very God of very God into the world would never have been used for any lesser purpose. This is why the idea that Jesus had step-siblings, or cousins, has gained acceptance. Some scholars, mostly Protestant, have suggested that Jesus very likely did have real blood brothers and sisters.
I do not think this issue is crucial to faith but if I had to draw a conclusion then I see, like Luther and Calvin, no good reason to reject the tradition of the church about Mary. One thing I am sure of—evangelicals desperately need to develop a much higher view of Mary. We can begin by stopping the reaction to tradition just because we do not agree with some of the added dogmas that came later in the development of Christian doctrine.