Praise be to Allah.
Praying in congregation is one of the greatest outward symbols of our faith. We have already stated that the most correct view, based on the evidence, is that it is obligatory. See question no. 120
With regard to the prayer of one who prays alone behind the last row, if there is space for him in the row, the scholars differed as to whether his prayer is valid. Imam al-Tirmidhi (may Allaah have mercy on him) said: Some of the scholars regarded it as invalid for a man to pray on his own behind the row. They said that he should repeat the prayer if he prayed behind the row on his own. This was also the view of Ahmad and Ishaaq.
He also narrated this view from Hammaad ibn Abi Sulaymaan, Ibn Abi Layla and Wakee’.
And he said: Some of the scholars said that his prayer is good enough if he prays behind the row on his own. This is the view of Sufyaan al-Thawri, Ibn al-Mubaarak and al-Shaafa’i.
The more correct view, which is that of Imam Ahmad and other scholars, is that it is not permissible for a person to pray on his own behind the row if it is possible for him to stand in the row; if he does that (i.e., stands on his own behind the row) then his prayer is invalid and he has to repeat it.
This is indicated by the hadeeth narrated by Abu Dawood (682), al-Tirmidhi (230) and others, according to which a man prayed on his own behind the row and the Messenger of Allaah (peace and blessings of Allaah be upon him) commanded him to repeat the prayer. Classed as saheeh by al-Albaani in al-Irwa’.
Al-Mubaarakfoori (may Allaah have mercy on him) said: This indicates that prayer offered standing on one's own behind the row is not valid, and that whoever prays on his own behind the row has to repeat his prayer. End quote.
That is also indicated by the hadeeth narrated by Ahmad (15862) and Ibn Maajah (1003), according to which ‘Ali ibn Shaybaan came in a delegation to the Messenger of Allaah (peace and blessings of Allaah be upon him). He said: we prayed behind the Prophet (peace and blessings of Allaah be upon him), and he saw out of the corner of his eye a man who did not make his back straight when he bowed and prostrated. When the Messenger of Allaah (peace and blessings of Allaah be upon him) had finished praying, he said: “O Muslims, there is no prayer for one who does not make his back straight when he bows and prostrates.” And he saw a man praying behind the row, so he waited until the man had finished, then the Messenger of Allaah (peace and blessings of Allaah be upon him) said to him: “Repeat your prayer, for there is no prayer for a man who stands alone behind the row.” Classed as saheeh by Saheeh Ibn Maajah.
Al-Sindi (may Allaah have mercy on him) said in his commentary on Ibn Maajah: The apparent meaning of this hadeeth is that the prayer of one who does that is invalid.
Al-San’aani (may Allaah have mercy on him) said:
This indicates that the prayer of one who prays behind the row on his own is invalid, because this is the apparent meaning of the hadeeth.
Al-Nakha’i and Ahmad were also of the view that such a prayer is invalid. Al-Shaafa’i regarded this hadeeth as da’eef (weak) and said: If this hadeeth were proven to be sound I would follow it. Al-Bayhaqi said: The best idea is to avoid doing that because this report is proven to be sound. End quote.
Shaykh Ibn ‘Uthaymeen (may Allaah have mercy on him) was asked: what is the ruling on praying behind the row on one's own?
He said: It is not permissible to pray behind the row on one's own and that prayer is not valid, according to the most sound scholarly view. This is the well known view of Ahmad (may Allaah have mercy on him), although there is another view reported from him which says that it is valid. The view that the prayer of a person who stands on his own behind the row is valid is the view of the other three imams, Maalik, Abu Haneefah and al-Shaafa’i.
But the more correct view is that it is not valid to pray behind the row on one’s own, unless there is no room to stand in the row. If the row is full and a person prays behind the row standing on his own and following the imam, then he is excused, and one is not obliged to do something that one is unable to do, as was stated by the scholars (may Allaah have mercy on them). Fataawa al-Shaykh, 15/193
In order for the one who does this action that has been asked about to understand the seriousness of it, you should know that some of the scholars are the view that the prayer of the one who prays on his own behind the row is invalid in all cases, even if he cannot find a space in the row. It says in Fataawa al-Lajnah al-Daa’imah: If a man enters the mosque and the iqaamah for prayer has been given and the row is full, he should try to enter the row. If he cannot do that then he should join the imam and stand on his right. If he cannot do that then he should wait for someone else to come with whom he can form a new row. If no one else comes then he should pray on his own after the congregational prayer finishes.
The same details were mentioned in another answer given by the Committee, in which it says that it is not sufficient to stand next to boys if they are under the age of discernment. They said: with regard to standing next to boys, if they have reached the age of discernment then standing next to them is valid, but if they have not reached the age of discernment then the ruling is the same as that on one who stands on his own behind the row, and the prayer of one who stands on his own behind the row is not valid, because the Prophet (peace and blessings of Allaah be upon him) said: “There is no prayer for one who stands on his own behind the row.” See Fataawa al-Lajnah, 8/6-7
Undoubtedly the situation which you have asked about goes completely against the wisdom behind prayer in congregation. Anyone who has the slightest understanding of the aims and purposes of sharee’ah would reject this. Shaykh al-Islam Ibn Taymiyah (may Allaah have mercy on him) said, after quoting the two ahaadeeth which are stated as evidence that the prayer of one who prays on his own behind the row is invalid:
More than one of the scholars of hadeeth have classed these two hadeeths as saheeh, and their isnaads are strong. Indeed, in many cases those who disagree with them rely on reports that have weaker isnaads than these two. There is nothing in these two hadeeths that goes against the basic principles of Islam, rather what is in them is what is implied by the well known texts and the basic principles, because prayer in congregation is called jamaa’ah because it brings together (ijtimaa’) the worshippers in doing something at the same time and in the same place. If they cancel out that unity of time or place, such as if all or some of them were to stand in front of the imam, or to keep far back from him with no excuse, that is forbidden according to scholarly consensus. Similarly if they were to be scattered instead of standing together, this is something which is most emphatically forbidden. Rather they are commanded to stand firmly and closely together, and the Prophet (peace and blessings of Allaah be upon him) commanded them to make the rows compact, straight and close together, and to fill all the gaps, starting from the front row. This is all aimed at joining them together in the best way possible.
If forming rows was not obligatory, then it would be permissible for one person to stand behind another and so on. Everyone knows that this is not the way that Muslims pray. If that were permissible then the Muslims would have done it, even if only once. The same would apply if they made the rows uneven, with one standing ahead of another and one standing back from another. But that is something that the Prophet (peace and blessings of Allaah be upon him) forbade.
If they pray in front of the imam, this is better than praying without organized rows. If the majority of scholars say that praying ahead of the imam is not correct, either in all cases or if there is no excuse, how could prayer be valid without having properly organized rows?
Study of the original sources shows that it is obligatory to pray in rows, and that the prayer of one who stands alone is not valid, as it says in these two hadeeths. If any scholar disagreed with that, then undoubtedly this Sunnah did not reach him in a trustworthy report; he may not have heard it at all, or he may have thought that the hadeeth was da’eef (weak), as some of them said. Majmoo’ Fataawa Shaykh al-Islam, 23/393-395
Try to offer advice to this brother who is doing this, and tell him of the Sunnah of the Prophet (peace and blessings of Allaah be upon him), and the fact that he forbade doing such things. But your advice must be given in a kind and gentle manner, choosing the circumstances in which you hope that your advice will be accepted. May Allaah help us and you to do that which He loves and which pleases him.