The 52-year-old, who won the World Cup as a player with Germany in 1990, had been in charge of the US since 2011.
US Soccer Federation president Sunil Gulati announced the "difficult decision" without naming a replacement. The United States does not face another World Cup qualifier until March.
"While we remain confident that we have quality players to help us advance to Russia 2018, the form and growth of the team up to this point left us convinced that we need to go in a different direction," Gulati said.
"With the next qualifying match in late March, we have several months to refocus the group and determine the best way forward to ensure a successful journey to qualify for our eighth consecutive World Cup."
Klinsmann went 55-27 with 16 drawn in a coaching tenure that began on July 29, 2011, when Gulati named him to replace sacked Bob Bradley on a team where Bradley's son Michael was a star midfielder.
"We want to thank Jurgen for his hard work and commitment during these last five years," Gulati said. "He took pride in having the responsibility of steering the program and there were considerable achievements along the way."
Klinsmann guided the Americans on a team record 12-match win streak in 2013 and his 16 triumphs and .761 win percentage that year set US calendar-year records.
But the Americans were a disappointing fourth in the 2015 Gold Cup and also lost a playoff to Mexico for a berth in the 2017 Confederations Cup.
Earlier this month, the US stumbled badly in opening the final round of North American (CONCACAF) qualifying for the 2018 World Cup in Russia, falling 2-1 at home to arch-rival Mexico and 4-0 at Costa Rica.
In a New York Times story published Monday, Klinsmann dismissed much of the criticism as "a lot of talk from people who don't understand soccer or the team" while saying critics were "ignoring the facts" and "being disrespectful."
Klinsmann has tinkered with lineups and formations but dismissed calls his moves hurt the squad, saying "we are coaching a team through a transitional phase."
Arena to return?
Gulati addressed the future of US Soccer, which many fans saw as bleak under Klinsmann's guidance, saying "we look forward to continued progress."
Bruce Arena, a five-time Major League Soccer champion coach now with the Los Angeles Galaxy, is thought to be a prime candidate to replace Klinsmann. The 65-year-old American coached the US team to the 2002 World Cup quarter-finals and guided a 2006 US lineup that exited in the group stages.
Sports Illustrated reported that Arena was set to take over and an announcement could come as early as Tuesday, when Gulati plans to speak with reporters.
Under Klinsmann in 2014, the Americans matched their 2010 World Cup showing by reaching the last 16 at Brazil after Klinsmann left long-time US striker Landon Donovan off the roster.
"Never want to see someone lose their job," Donovan tweeted Monday. "I have no doubt that this talented group of players will qualify for Russia '18."
Gulati sang Klinsmann's praises even as he swung the axe, saying the German legend's legacy will have a long-term impact on the squad.
"Many are aware of the historic victories, including leading us out of the Group of Death to the round of 16 in the 2014 FIFA World Cup, but there were also lesser publicized efforts behind the scenes," Gulati said.
"He challenged everyone in the US Soccer community to think about things in new ways and thanks to his efforts we have grown as an organization and expect there will be benefits from his work for years to come."
Eight US matches remain in the North American (CONCACAF) "hexagonal" round-robin qualifying, the next being at home to Honduras on March 24 and four days later at Panama.
The three top teams in the six-nation field advance to the World Cup while the fourth faces a playoff against an Asian team for one of the last berths in Russia.