Praise be to Allah.
Does masturbation invalidate hajj?
Masturbation is haram during Hajj and otherwise, but during ihram it is worse and more serious, although it does not mean that hajj is rendered invalid according to the majority of jurists, with the exception of the Malikis.
According to the majority, a compensatory sacrifice is required for that, which is a camel according to the Hanbalis. However, according to the view that it renders Hajj invalid, it is still obligatory to continue with it, and also make it up the following year. (See: al-Mawsu’ah al-Fiqhiyyah, 2/193) and al-Insaf , 3/224)
The correct view is that of the majority, which is that masturbation does not render hajj invalid , because there is no evidence to suggest that it does so, and because the analogy comparing masturbation to intercourse is weak.
Masturbated in hajj: Should I complete the rituals?
You made a much more serious mistake by not completing your Hajj, even according to the view that it was rendered invalid, because it was still obligatory to complete it. If you did not complete it, then you are still in a state of ihram, according to both views.
Ad-Dardir said in ash-Sharh al-Kabir ma‘a ad-Dasuqi (2/68):
“It is haram for both spouses to have intercourse or do the things that lead to it… Having intercourse renders Hajj and `umrah invalid in all cases , regardless of whether it was done by mistake or he forced her, and whether he did any of the actions of Hajj after entering ihram or not, and whether he had reached the age of puberty or not, such as doing anything that leads to the emission of maniy. It is haram and renders Hajj invalid if emitted.”
Then he said:
“It is still obligatory – with no difference of opinion among the scholars concerning that except Dawud – for the one who has rendered it invalid to complete that Hajj or `umrah , so he should continue with it, like a valid Hajj, if he manages to stand in `Arafah. But if he cannot stand in `Arafah due to some obstacle or impediment, then he must exit ihram by doing `umrah , and it is not permissible for him to remain in ihram until the following year, because by doing that he is persisting in a Hajj that is invalid when he is able to extricate himself from it. If he could not complete his Hajj, whether he thought it was permissible to interrupt it or not, then he is still in a state of ihram, even if he renewed his ihram with the intention of making up (for that invalidated Hajj) or not, and that second ihram does not count for anything. As he is still in a state of ihram for the first Hajj, and he entered ihram the following year to make up for that Hajj, that cannot make up for the invalidated Hajj, and whatever he does the following year is to be regarded as completing the invalidated Hajj; making it up cannot be done until the third occasion, if it was `umrah , or the third year if it was Hajj.”
It says in ash-Sharh as-Saghir ma‘a as-Sawi (2/95):
“The one who invalidated his Hajj by having intercourse or ejaculating – whether he thought it was permissible to interrupt it because it had been spoiled or not – remains in a state of ihram for as long as he lives.”
Validity of rituals done while in a state of ihram because of not completing hajj
Based on the view of the majority, your Hajj was not invalidated by masturbating, and you should have completed it. If you did not do tawaf al-ifadah, or the sa`i of Hajj, or you did not intend to do one of them, then you are still in a state of ihram and your doing `umrah did not free you from ihram; rather this `umrah does not count, according to the majority of scholars, or you may be regarded as doing qiran because you incorporated it with Hajj, as is the view of the Hanafis. (See: al-Mawsu‘ah al-Fiqhiyyah (2/140); ash-Sharh al-Mumti‘ (7/86)
You must go to Makkah and do tawaf and sa`i, and thus you will exit ihram of Hajj.
If you had stoned the Jamarat without forming the intention, then you must offer a compensatory sacrifice for that, because stoning the Jamarat without forming the intention comes under the same ruling as not doing it at all.
A marriage contract done whilst still in a state of ihram is not valid, so it must be redone after exiting ihram.
What you did during this period of actions that are forbidden whilst in ihram is overlooked, regardless of whether you had completed the first stage of exiting ihram or not; you are excused because you were unaware of the ruling.
Not remembering a condition when entering ihram: does it count?
With regard to what you mention about stipulating a condition when entering ihram [that if some impediment prevented you from continuing and completing Hajj, you would exit ihram at that point with no penalty], if you are not certain that you said it, it does not count, because the basic principle is that the condition was not stipulated, unless you remember that you did it.
Rather what appears to be the case is that stipulating a condition is of no benefit in this instance, because there was no impediment that prevented you from completing the rituals.
What you must do is repent to Allah, may He be exalted, from what you have done and from your heedlessness in not asking and finding out what you needed to know about your acts of worship.
We ask Allah to accept from us and from you.
And Allah knows best.