Mori drew fire from figures including politicians joker game and sports stars, and his apology backfired after he appeared to defend his remarks and told reporters: "I don't speak to women much."
Several hundred Olympic volunteers have withdrawn since the comments, which prompted a barrage of complaint calls as well as a petition calling for action against Mori that has gathered nearly 150,000 signatures.But while the 83-year-old former prime minister now appears headed for the exit, his reported choice for a successor, Saburo Kawabuchi, has raised eyebrows.
A year older than Mori, Kawabuchi is a long-time sports administrator and former professional footballer who competed when Tokyo last hosted the Olympics in 1964.
"Inside the organising committee, there are some voicing concern," the Asahi Shimbun newspaper reported.
"I don't think an old man like him taking over will convince the public," it cited a source involved in organising the Games as saying.
"I think the meeting on Friday could see some debate," the source added.
Kawabuchi is considered an able and well-connected administrator, and currently serves in the symbolic role of mayor of the Olympic Village.However, the appearance of Mori hand-picking his successor has reportedly not gone down well.
"It makes no sense for a resigning chief to appoint his successor. There are steps to this process. If this is allowed, there's no point to even having the meeting," one Tokyo 2020 board member told the Mainichi Shimbun.
- Fresh headache for organisers -
Kawabuchi has said he wants Mori to remain a key adviser, adding further cause for concern, the Mainchi said.
Mori "left a mechanism to maintain his influence by confirming his own successor ahead of the emergency meeting on the 12th," the paper wrote.Olympic Minister Seiko Hashimoto told reporters Friday that "nothing has been decided" on Mori's successor.
"The organising committee will make a decision... while listening to opinions from a range of people," she said.
"It is desirable to go through a formal procedure."
The row comes with organisers already facing public doubt about holding the huge international event this summer.Around 80 percent of Japanese polled in recent surveys back either a further postponement or an outright cancellation.
Organisers have tried to quell the disquiet by releasing virus rulebooks for athletes, officials and media, including restrictions on movement and regular testing.But with Tokyo and other parts of the country under a virus emergency, doubts persist about the viability of the Games.
The first Olympic test event of the year has already been postponed because of Japan's current strict virus entry rules, and Tokyo has yet to approve a single vaccine.