Praise be to Allah.
With regard to that which is known on the basis of Islamic teachings to be invalid because an essential part of it was omitted, or one of the conditions of it being valid was not met, or because the doer did something that rendered it invalid, and so on, then we may be certain that it is not accepted, such as someone who prays before the time for that prayer begins, or who prays but does not recite al-Faatihah, or who deliberately eats and drinks during the day in Ramadan. These and similar cases are known to be invalid on the basis of Islamic teachings, so stating definitively that they are not accepted is valid. But if the worshipper fulfilled all the conditions of prayer and did all the essential parts, and he did not do anything obvious that could render it invalid, then in this case no one can say whether his prayer is accepted or not, because that is based on what he has in his heart of sincerity and devotion to Allah, and this is something that no one can know except Allah, may He be exalted.
The scholars of the Permanent Committee for Ifta’ said:
Knowledge of whether an act of worship was accepted or not is a matter of the unseen, which no one knows except Allah.
End quote from Fataawa al-Lajnah ad-Daa’imah (12/195).
If someone says: So-and-so’s prayer is not accepted:
if he said that because he saw him do something in his prayer that rendered it invalid, such as if he knows that he prayed in a state of impurity, or that he omitted one of the essential parts of the prayer, or he did something that invalidated the prayer, then what he said is correct, and there is no blame on him.
But if he did not know of anything to imply that his prayer was invalid; rather he said that because he saw him doing some evil action or indulging in some foul talk and the like, so he said definitively that his prayer was not acceptable, this is haraam and is not permissible, and it comes under the heading of saying about Allah something that is not true, because whether prayer is accepted or not is something that no one knows except Allah, and to say something about Allah with no basis is a major sin. But it does not come under the heading of swearing an oath by Allah about Allah, and in this case he did not swear an oath to that effect.
Muslim (2621) narrated from Jundub that the Messenger of Allah (blessings and peace of Allah be upon him) narrated that “A man said: ‘By Allah, Allah will not forgive So and so.’ Allah said: ‘Who is the one who swore by Me that I will not forgive So and so? I have forgiven So and so and I have cancelled out your good deeds.’”
See the answer to question no. 14258 to learn about the conditions of deeds being acceptable to Allah, may He be glorified and exalted.
And Allah knows best.