Sheikh Mohammad Ahmed Hussein, who decried the mosque closure, was taken into custody from the Bab Al-Asbat area (Lion's Gate) after leading an open-air prayer nearby.
"Israeli police detained my father in a violent manner and took him to unknown destination," said Omar, the grand mufti's son.
Hussein was later released on $2,800 bail.
Al Jazeera's Harry Fawcett, reporting from Jerusalem's Old City, said it was the first time in decades that the compound closed for prayers on Friday.
"Of course this has the potential to increase tensions among the about 10,000 Palestinians who normally come here for worship," Fawcett said.
Sheikh Omar Keswani, a religious official at Al-Aqsa, decried the closure and cancellation of Friday prayers.
"Forbidding the Friday prayer is an unfair procedure," Keswani told Al Jazeera. "What happened earlier is now being taken advantage of by the Israeli right to impose a new reality in Al-Aqsa mosque."
Haj Khalil Abu Elsheikh, 77, travelled 100km from Beer-sheva to attend Friday prayers at Al-Aqsa mosque.
"No religion allows this," he told Al Jazeera, referring to praying on the side of the road. "No belief accepts this."
The rare attack from within the contested Al-Aqsa compound, revered by both Muslims and Jews, raised new concerns about an escalation of violence.
The ancient marble-and-stone compound - known to Muslims as al-Haram al-Sharif - houses the Al Aqsa mosque, Islam's third-holiest site, and the 7th century Dome of the Rock. Thousands pray there every Friday.
The Western Wall of the compound, also known as the Wailing Wall, is considered the holiest site in Judaism.
Al Jazeera's Fawcett said the Palestinians reportedly shot at Israeli security forces near the Lion's Gate to the Old City.
"The three attackers were armed with two machine guns, a pistol and a knife, according to Israeli police," Fawcett reported.
"They were then pursued inside the Al-Aqsa mosque compound. In the courtyard of the compound a final gun battle ensued between the gunmen and Israeli security forces.
Following the attack, Israeli police cleared the Al-Aqsa mosque and closed it for public. They identified the assailants as Arab citizens of Israel.
Israeli Prime Minister Benjamin Netanyahu quickly tried to allay Muslims' fears, saying that the status quo at the Muslim-administered site "will be preserved".
Palestinian President Mahmoud Abbas talked to Netanyahu in a rare phone call to discuss the situation in Jerusalem, highlighting the concern about a possible escalation, according to the official Palestinian news agency WAFA.
Abbas condemned the attack and said he rejects "any violence from any party, particularly at holy sites".
Netanyahu has said the compound would be gradually reopened on Sunday.
Meanwhile, Jordan's government spokesman Mohammad Al Momani called on Israel to end the closure, which he described to Petra news agency as an "attack on the right of Muslims to practise their religious rituals in their holy sites".