The International Olympic Committee (IOC) has said it sees no reason to delay or move the Games because of the mosquito-borne disease.
It said this would "not significantly alter" the spread of the virus, which is linked to serious birth defects.
In an open letter to the WHO, more than 100 leading scientists had said new findings about Zika made it "unethical" for the Games to go ahead.
They also said the global health body should revisit its Zika guidance.
The outbreak began in Brazil a year ago, but now more than 60 countries and territories have continuing transmission.
While Zika's symptoms are mild, in the letter the experts say it causes babies to be born with abnormally small heads and may also cause a rare and sometimes fatal neurological syndrome in adults.
The letter is signed by 150 international scientists, doctors and medical ethicists from such institutions as Oxford University and Harvard and Yale universities in the United States.
They cite the failure of a mosquito-eradication programme in Brazil, and the country's "weakened" health system as reasons to postpone or move the Olympics in "the name of public health".
"An unnecessary risk is posed when 500,000 foreign tourists from all countries attend the Games, potentially acquire that strain, and return home to places where it can become endemic," the letter says.
The biggest risk, it adds, is if athletes contract the virus and returned home to poor countries that have not yet suffered a Zika outbreak.
They also express concern the WHO has a conflict of interest because of its partnership with the IOC.
The Rio Olympics are due to take place between 5 and 21 August.
In a statement, the WHO, which has declared the Zika virus a global public health emergency, said: "Brazil is one of almost 60 countries and territories which to date report continuing transmission of Zika by mosquitoes.
"People continue to travel between these countries and territories for a variety of reasons. The best way to reduce risk of disease is to follow public health travel advice."
Several public health experts had previously warned that hundreds of thousands of people arriving in Rio would speed up Zika's spread and lead to the births of brain-damaged babies.
But on Thursday, the head of the US Centers for Disease Control and Prevention (CDC), whose advice is quoted approvingly in the letter, said the threat did not warrant halting the Games.
"There is no public health reason to cancel or delay the Olympics," Dr Tom Frieden said.
However, he urged the US to act more quickly to prevent pregnant women contracting Zika, amid congressional deadlock over the release of $1.9bn (£1.3bn) in funding.