The club agreed to allow women members at an extraordinary board meeting on Monday.
The Kasumigaseki Country Club in Saitama prefecture, north of Tokyo, agreed to allow women members at an extraordinary board meeting on Monday, JiJi Press said.
Immediate comment from the club was not available, but the decision was welcomed by organisers of the Tokyo Games.
It came after IOC chief Thomas Bach on Friday reiterated his threat to take the golf event elsewhere, saying he was hopeful the club would opt for "non-discrimination".
Tokyo's female governor Yuriko Koike had also complained about the ban, saying she felt "extreme discomfort" to see women with no right to be full members in the 21st century.
The Japanese club has women on its roster but they are not allowed to become full members and cannot play on certain Sundays -- restrictions that do not apply to male members.
Tokyo 2020 chief Yoshiro Mori welcomed the club's decision, saying it amended the policy "in keeping with the spirit of the Olympic Charter."
"I truly appreciate the numerous efforts that the club's senior leaderships and all the club members have made so far to meet the requirements for hosting Olympic competitions," Mori said in a statement.
The club's chairman told local media earlier this month he was "flummoxed" by the row, prompting further criticism.
The decision also came after Scotland's prestigious Muirfield club voted to allow women members last week, ditching a rule which had stood for 273 years.