He let him off the debt and now he is asking for it to be paid

Praise be to Allah.

If the lender lets the debtor off from paying the debt, then he no longer owes the money, and if he no longer owes the money he cannot be asked for it, unless he has borrowed money again. 

Based on this it is not permissible for you to ask for the money back after you let him off, and you have no right to the money that he gave you. You have to return the money to him. 

The Standing Committee was asked about a man who let off the person who owed him money, then he asked for it and took it from him. They replied: 

If the matter is as described, you do not have the right to take the money that you already let him off paying.  

Fataawa al-Lajnah al-Daa’imah, 13/177 

But if you told him (that you have no right to it) but he still let you keep the money, there is nothing wrong with taking it. 

And Allaah knows best.