The Glasgow giants looked to be heading to their second successive draw in the Scottish Premiership until substitute Forrester popped up in the third minute of injury time to head home fellow sub Joe Dodoo's cross.
The win - only their sixth of the season - means third place Rangers reduce the gap with league leaders Celtic to 11 points and remain one point behind second place Aberdeen who came from behind to secure a 3-1 win away to Inverness.
Rangers manager Mark Warburton said the win was well-deserved.
"They left it late but I thought we were good for long periods," Warburton said.
"They stuck to their task. They could sense the frustration but what they have to do is believe in what they do and they more than got their rewards today."
Defeat sees Dundee slip to the bottom of the table after Ross County defeated St Johnstone 3-1 to end a 10-match winless run.
Dundee manager Paul Hartley said a momentary lapse of concentration cost his side dearly.
"We were a minute away from getting a thoroughly-deserved point but we need to concentrate for the whole game," Hartley said.
"We defended brilliantly for the majority of the game but it was a real killer blow for the players because I thought they were terrific today."
Rangers kept faith with the same starting 11 that were held to a disappointing draw with Ross County before the international break and the Gers players looked like they were desperate to make amends for their drab display in Dingwall as they dominated the early proceedings.
The first real chance fell to Joe Garner after he collected a Josh Windass pass on the edge of the box before he spun and flashed a low shot narrowly wide.
The English striker then sent an ambitious lobbed shot on to the roof of the net after catching Dundee 'keeper Scott Bain off his line.
Garner, who has enjoyed a stuttering start to his Ibrox career following his summer move from Preston, was clearly in the mood and turned provider as he launched a dangerous ball across the face of the Dundee goal that veteran striker Kenny Miller was close to converting.
Dundee finally came to life in the 25th minute as Wes Foderingham was forced to scramble low to his right to push a dipping 25-yard free-kick from Tom Hately round the post. From the resultant corner Marcus Haber was handed a free header at the far post but his effort was cleared off the line by the Rangers defence.
Rangers started the second half on the front foot and Garner was inches away from connecting with Windass' delivery from the right. The English striker wasn't far away moments later when he tried his luck with a long range effort.
Suddenly Dundee sensed an opportunity and Kevin Gomis should have done better after he was left unmarked at a free-kick but he headed well over before Jason Holt made a crucial goal-saving intervention to deny Haber.
Rangers rallied but Garner's luck in front of goal failed to improve with the forward again not able to get on the end of substitute Joe Dodoo's cross from right.
In what has become a familiar scenario this season, Rangers dominated possession but could not find a way to convert their chances and breach a well-organised and determined Dundee defence and seemed to be heading to the sixth draw of the season.
However, with time running out Dodoo swung the ball deep into the Dundee box from the left and Forrester sent a backward header into the bottom right hand corner of Bain's net, much to the relief of the home support.