Andy Johnson had looked set to fire Palace to a vital win when he nodded home his 15th goal of the season two minutes into the second half.
But that only served as the prelude to amazing final eight minutes when Albion drew level through Kevin Campbell and then took the lead thanks to Robert Earnshaw, before seeing Riihilahti salvage a share of the spoils for Palace.
Palace's point was all the more worthwhile as they had to play for 78 minutes of the game with ten men after the dismissal of central defender Gonzalo Sorondo for a foul on Geoff Horsfield.
Even though Horsfield appeared to be heading away from goal, referee Dermot Gallagher had no hesitation in dismissing Sorondo.
Palace boss Iain Dowie took off striker Dougie Freedman and sent on Darren Powell who marshalled the defence superbly.
They produced a tremendous battling display to keep Albion at bay before Johnson pounced.
The goal was a personal disaster for the Albion skipper and Johnson's former Birmingham City team-mate, Darren Purse.
Purse, who had warned before the game about the threat Johnson posed, ducked under a long clearance from Palace goalkeeper Gabor Kiraly in order to let the ball bounce through to his own keeper, Russell Hoult.
But Johnson, who scored twice against Albion when Palace beat them 3-0 at Selhurst Park in October, nipped in behind him to head the ball over Hoult and into an empty net.
Albion struggled to bounce back from the shock of falling behind and it was not until the final ten minutes that they really built up a head of steam.
Zoltan Gera whipped in an 82nd minute cross that was sliced towards his own goal by central defender Powell. Palace goalkeeper Kiraly somehow managed to prevent an own goal with a superb one-handed save.
But the loose ball fell to the alert Campbell who bundled the ball home for his second goal since his arrival from Everton last month.
Albion had Palace on the rack at this stage and as the game moved into stoppage time, Earnshaw looked to have won the game with his eighth goal of the season.
The Welsh international, who had replaced Horsfield in the 55th minute, was sent clear by Campbell and lifted a rising shot over Kiraly.
It looked to be a decisive strike, but Albion were undone in the closing seconds of the match when a long ball into the area caused panic in their defence.
Mikele Leigertwood headed the ball back into a crowded goalmouth and Riihilahti - on the pitch for just four minutes - nodded the ball into the back of the net with Fitz Hall at the back post ensuring that the ball crossed the line.
Palace's late heroics cost Albion the chance to move off the bottom of the table and also reduce the five-point gap between themselves and the Selhurst Park side.