Praise be to Allaah.  

Firstly: 

If this loan is based on riba (usury, interest), then it is haraam and is one of the major sins and one of the seven sins that doom a person to Hell. All of the nations regarded it as forbidden, even the Greeks who were idol-worshippers. One of them, whose name was Solon said: Money is like a sterile hen; a drachma cannot give birth to a drachma. 

In the Christian belief it says that one who consumes usury should not be shrouded if he dies, and even the Jews forbade usury. 

Islam has forbidden it in a manner that leaves no room for doubt concerning its prohibition.  

Allaah says (interpretation of the meaning): 

“Allaah has permitted trading and forbidden Ribaa. So whosoever receives an admonition from his Lord and stops eating Ribaa, shall not be punished for the past; his case is for Allaah (to judge); but whoever returns (to Ribaa), such are the dwellers of the Fire — they will abide therein”

[al-Baqarah 2:275] 

“O you who believe! Fear Allaah and give up what remains (due to you) from Ribaa (from now onward) if you are (really) believers”

[al-Baqarah 2:278] 

Abu Juhayfah (may Allaah be pleased with him) narrated that the Messenger of Allaah (peace and blessings of Allaah be upon him) forbade the price of blood, the price of a dog and the earnings of a prostitute, and he cursed the woman who does tattoos and those who have them done, the one who consumes riba (usury) and the one who pays it, and he cursed the one who makes images. 

Narrated by al-Bukhaari, 2123. 

It was narrated that ‘Abd-Allaah ibn Mas’ood (may Allaah be pleased with him) said: The Messenger of Allaah (peace and blessings of Allaah be upon him) cursed the one who consumes riba and the one who pays it. 

Narrated by Muslim, 1597. 

It was narrated from Abu Hurayrah (may Allaah be pleased with him) that the Prophet (peace and blessings of Allaah be upon him) said: “Avoid the seven sins that doom a person to Hell.” They said, “O Messenger of Allaah, what are they?” He said: “Associating others in worship with Allaah (shirk); witchcraft; killing a soul whom Allaah has forbidden killing, unless that is done lawfully; consuming riba (usury, interest); consuming the property of orphans; running away from the battlefield; slandering innocent chaste believing women.” 

Narrated by al-Bukhaari, 2615; Muslim, 89 

It was narrated that Samurah ibn Jundub (may Allaah be pleased with him) said: The Prophet (peace and blessings of Allaah be upon him) said: “Last night I saw (in a dream) that two men came to me and took me out to a sacred land. They set out with me until we came to a river of blood in which a man was standing, and on the bank of the river there was a man with stones in front of him, facing the man who was in the river. Every time the man wanted to get out, the man (on the bank) threw a stone into his mouth and he went back to where he had been. I said: ‘What is this?’ They said, ‘The one whom you saw in the river is the one who consumed riba.’” 

Narrated by al-Bukhaari, 1979. 

So you have to repent to Allaah from this action. 

But if the loan was a good kind of loan that did not involve riba, there is nothing wrong with it. 

Secondly: 

With regard to Hajj: the person who cannot spend on himself because he is of limited means is not obliged to go for Hajj, but which of the two is more important, going for Hajj or paying off the debt? 

The most correct view is that paying off the debt is more important, because the debtor is not obliged to do Hajj, since one of the conditions of Hajj is that one should be able to do it. 

If there is a conflict between your doing Hajj and your paying off your debt, then give priority to paying off the debt. But if there is no conflict, such as there being plenty of time for you to pay off your debt, or if the person to whom you owe money is patient and can wait, then the correct view is that there is nothing wrong with your doing Hajj or ‘Umrah. 

Shaykh al-Islam Ibn Taymiyah (may Allaah have mercy on him) said: 

It is permissible for a person who is in debt and facing financial hardship,  either because he is unable to earn money or because the creditor is absent and he cannot pay him, to go for Hajj if someone else takes him for Hajj, provided that this does not affect his ability to pay the debt. 

Majmoo’ al-Fataawa, 26/16 

All of that is subject to the condition that you are completely able to perform Hajj and that you pay off debts to anyone who asks you to pay him back when the debt becomes due, and that you are able to pay for the journey and its provisions, and whatever you need to take care of your affairs whilst you are travelling, without neglecting your family or those on whom you are obliged to spend. 

So you should leave with them enough to cover their needs. If you do not do that then you are sinning and neglecting those whom Allaah has commanded you to take care of. 

It was narrated that Khaythamah said: we were sitting with ‘Abd-Allaah ibn ‘Amr when Qaharmaan came to him. He said, ‘Have you given the slaves their food?’ He said, ‘No.’ He said, ‘Go and give it to them. The Messenger of Allaah (peace and blessings of Allaah be upon him) said: “It is sufficient sin for a man if he withholds the food of his slave.”’ 

Saheeh Muslim, 996 

It was narrated from Ibn ‘Umar that the Messenger of Allaah (peace and blessings of Allaah be upon him) said: “It is sufficient sin for a man if he neglect those whom he is responsible for.” 

Narrated by Abu Dawood, 1692 

And Allaah knows best.