The ceremony comes after his predecessor, Pierre Nkurunziza died suddenly last week following a cardiac arrest on June 8, 2020.
AFP reports indicated that attendees were asked to arrive on time so they could practice some COVID-19 precautions like hand-washing and temperature checks.
Mr Ndayishimiye is a former rebel leader, like Mr Nkurunziza.
His deceased predecessor backed him during the election which he won in May. However, the opposition said the election was rigged.
Mr Nkurunziza has been in power for 15 years and was to step down in August.
The constitution of Burundi states that if the head of state dies in office then the president of the national assembly, currently Pascal Nyabenda, should succeed him.
However, a ruling by Burundi's Constitutional Court caused them to bring Mr Ndayishimiye's inauguration in the capital, Gitega, forward.
Mr Ndayishimiye has become president at a time when Burundi is diplomatically isolated and on bad terms with donors.
His predecessor was accused of leading a government that violated human rights and cracked down on the opposition, journalists, and activists.
Burundi is one of the world’s poorest countries and has also lost donor funding after the political violence following the 2015 election.
Thousands of Burundians have fled the country and into refugee camps following the failed coup attempt in 2015.
The country is also dealing with the coronavirus pandemic.
Its GDP per capita was $270 per person in 2018, World Bank statistics show.