Praise be to Allaah.
Before replying to this question, I would like to warn our brothers whom Allaah has blessed in this country with abundance of wealth and good things against persistently bringing female servants in, because this is a kind of luxury and is indeed a kind of extravagance. We even hear of some cases where there is no one but a man and his wife at home, and the woman is able to take care of all the household affairs, but they still bring a female servant. I warn my brothers against this serious problem which has become a matter of competition among people, where a wife will say that she wants a servant so the husband goes and brings her a servant. My advice is that no-one should bring a servant except in cases of extreme necessity.

My view is that even in cases of necessity, only Muslim servants should be brought in, because the Prophet (peace and blessings of Allaah be upon him) commanded that the Jews and Christians be expelled from the Arabian Peninsula. If a servant is brough in, she should not be young and pretty, because she is a source of temptation, especially if there are young men in the family. For the Shaytaan flows through the son of Adam like blood. And no female servant should be brought without her mahram, because the Prophet (peace and blessings of Allaah be upon him) forbade women to travel without their mahrams.

If the servant has a mahram, then the problem mentioned in the question does not arise; her mahram can go for Hajj with her. But if she does not have a mahram, or if her mahram brought her then went back, then they should not take her for Hajj or travel with her; rather she should stay with someone who can be trusted to take care of her. If there is no one who can be trusted to take care of her, then she may go for Hajj with them because of necessity, and her Hajj will be valid.
And Allaah is the Source of strength.