Robbie Muirhead scored a double to down the Glasgow giants and give head coach Robbie Neilson, who is expected to become English League One club MK Dons' new manager subject to the two clubs agreeing compensation, the perfect send-off.
The result sees Hearts leapfrog Rangers into second spot on goal difference following their third defeat of the season as they struggle to adapt to life in the top flight following a four-year absence.
The future of Warburton, whose side have dropped points in eight of their 15 games, is set to come under scrutiny with his side now trailing league leaders Celtic by eight points having played three games more.
And the former Brentford boss, who has seen the Gers record just one win against sides in the top six, admitted the Ibrox club's record against teams in the top half of the table has to improve quickly.
"Of course it does. That's an obvious one. You have to win your games of football," Warburton said.
"We were second going into tonight's game so we must be doing something right.
"But the fact of the matter is we had a poor performance tonight. There is no glossing over that.
"You've got to come to this places and win your battles, you've got to win your one v ones and you've got to do what you do best.
"Talk is cheap. We're professionals. All of us have to come here and be better.
"Tonight we came and we were well prepared, we were in good shape and we didn't deliver the performance we were looking for."
Rangers thought they had equalised in the 52nd minute when Joe Dodoo fired a sensational strike into the net. However, a very late offside flag from the assistant referee, who ruled Harry Forrester had interfered with play while in an offside position, saw the Rangers celebrations come to an end.
The delayed decision incensed Rangers boss Warburton, although he admitted it looked like it was the correct call.
"What caused us anger was that we just watched it again there and the flag has never gone up. We've gone away and celebrated and the flag's been down," Warburton fumed.
"The Hearts guy has gone over to talk to him and some 15 or 20 seconds after the event the flag goes up.
"It was inexplicable to us and we can't understand it. We're told that the linesman was debating it but it's either offside or it's not. The job of the linesman is to be in line with the last player and flag it.
"If it's offside it's offside but as far as our players are concerned the ball has gone in, there's no reaction so they've celebrated accordingly.
"I think Harry looked offside to be honest with you but the fact is the flag has not gone up, we've gone away and nothing has happened so I'm bemused by this decision."
A statement on Hearts head coach Neilson is expected Thursday but the 36-year-old was elusive about his future at Tynecastle after the game.
"I was absolutely delighted with the performance. It was 11 guys out there who gave everything for the club which was the most pleasing thing," Neilson said.
"I'll be in tomorrow morning as usual and we'll talk about things. I expect it to be clarified then. Tonight is all about the players.
"It's a phenomenal place. There's nowhere else where the fans back the team like this and I've been lucky to be on the journey."