After Real coach Zinedine Zidane called for Benzema's four-year France exile, caused by his alleged part in an attempt to blackmail Mathieu Valbuena, to end this week, Le Graet told French radio Benzema's adventure with the French team was finished.
"Noel, I thought you didn't interfere with the decisions of the France coach," said Benzema on Twitter.
"Please understand that I and I alone will call an end to my international career."
Benzema, who is of Algerian origin, added that Le Graet should "let me play for one of the countries for which I'm eligible" if he thinks the 31-year-old is done.
Benzema has not played for France since 2015, when he was placed under formal investigation over the blackmail attempt.
Coach Didier Deschamps has maintained Benzema's omission is in the best interests of the French squad, who won last year's World Cup without him.
Benzema has scored 9 goals in 11 La Liga appearances for Madrid already this season.
"He has always wanted to play for France," Zidane said this week.
"He is the best and I think he should definitely have a place in the national team. Football-wise, there is no debate."