Praise be to Allah.
If someone removes a bit of skin from his lip whilst fasting, he must spit it out.
If he swallows it by mistake (because he forgot), or he could not find it, or it is too difficult for him to take it out of his mouth, so he swallows it with his saliva, his fast is valid and he does not have to do anything, because even if we assume that it was still there and he swallowed it, he swallowed it without meaning to.
It says in Sharh Muntaha al-Iraadaat (1/483):
If he wakes up and finds a bit of food in his mouth, and he spits it out, or it is too difficult for him to spit it out, so he swallows it with his saliva without meaning to, that does not invalidate his fast, because it is too difficult for him to avoid that.
But if it can be distinguished from his saliva, and he swallows it deliberately, then he does break the fast. This was stated by Imam Ahmad. End quote.
So long as you did not deliberately swallow this bit of skin, then there is no blame on you, and you did not have to exit your prayer and rinse out your mouth; rather it is haraam to exit the prayer for that purpose; rather what you should do is remove it from your mouth with your finger or a handkerchief and the like.
And Allah knows best.