Cox, a £1.9m summer signing from Swindon Town, pounced from 12 yards in the 117th minute after cashing in on the persistence of Chris Wood.
It was tough on League Two Rotherham who had twice battled back from a goal down to take the game into extra time when they took the lead themselves.
Albion handed a comeback to Roman Bednar following his drug ban and he was in the thick of the early exchanges, setting up Rotherham old boy Reuben Reid for an eighth-minute shot that was blocked by 'keeper Andy Warrington before he played an inadvertent role in Albion taking the lead.
When Beattie curled in a cross shot from the left wing, the presence of Bednar on the edge of the six yard box was enough to put off Warrington and he was rooted to his line as the ball nestled in the far corner of the net.
Bednar should then really have doubled Albion's advantage after being set-up following good work between Cox and Reid only to see his close range drive deflected over the bar.
It proved a costly miss as Rotherham were on level terms just six minutes later.
Winger Nicky Law got away from right-back Shaun Cummings and delivered a perfect cross that Cummins powered past Dean Kiely.
Having been rocked by that setback, Albion were able to regain the initiative six minutes into the second-half when Bednar set up Dorrans who made no mistake with a rising drive from the edge of the area.
But Rotherham, who beat Derby County in the previous round, are nothing if resilient and this time they were only behind for little more than 60 seconds with Law again the provider.
His corner caught the Albion defence flat-footed and Tom Pope rose the highest to give Kiely no chance with a perfect header that ensured the game would move into extra-time.
Pope looked like he had won the game for Rotherham in the 98th minute when Leon Barnett failed to fully clear a cross from Ryan Taylor.
Pope was on to the ball in an instant and gave Kiely no chance with a fierce angled drive from the left hand edge of the area.
Having lost the lead twice themselves, this time it was down to Albion to show their fighting spirit and they did just that three minutes later when Beattie struck from the edge of the area after good work by Cox who then completed the comeback himself.