The flying winger had never scored for the Reds prior to the clash but trebled his tally with three goals in 14 second-half minutes to take the plaudits.
It was no more than Billy Davies' side deserved as they eased past the third round for the first time since 2011 to set up a tie at home to either Ipswich or Preston.
And they doubled their tally just after the hour when Paterson fired a low shot into the corner.
The winger then made it three with a deflected shot before sealing his hat-trick with a thumping drive after a fine solo run.
Andy Reid put the icing on the cake when he lashed into the roof of the net in stoppage time as an under-strength West Ham, fielding three young debutants, were put to the sword.
PRE-MATCH: Strong Forest side, youthful Hammers one - Stewart Downing captains a youthful Hammers side that includes three debutants. Danny Whitehead, Callum Driver and Sebastian Lletget all start, with new recruit Jaanai Gordon on the bench. Meanwhile, Billy Davies makes just two changes to the team that drew at Reading. Guy Moussi and Djamel Abdoun replace Nathaniel Chalobaha and Jamie Mackie.
24' - GREAT SAVE. Ravel Morrison steps up and curls a delightful effort towards the top corner, but Karl Darlow scurries to his right and manages to tip it over the bar. 42' - QUICKFIRE FOREST CHANCES! Lichaj ghosts into the area but side foots his shot straight at Adrian, who makes a good save. The ball rebounds out to Lansbury who takes a touch before lashing it over the top from 18 yards out. 59' - WEST HAM CHANCE. Mayhem in the Forest defence. A Downing free kick is only half-cleared, allowing Lletget to fizz in a low shot that Darlow just about keeps out. The ball is clipped back in and forest just about survive. 65' - GOAL: Forest 2-0 West Ham. Halford scraps for possession on the edge of the box, allowing Paterson to burst onto the loose ball and fire a low shot across goal and in off the far post. 68' WHAT A MISS! Abdoun breaks free down the left and tees up Paterson who crashes a shot on the stretch well over. 71' GOAL! Forest 3-0 West Ham. Paterson grabs his second of the game with a deflected, low finish after Lansbury dummied a shot and teased a pass into his path. 79' GOAL! Hat-trick! Paterson jinks beyond three challenges and then fizzes a low shot into the corner to seal his treble in style. 4-0 and it's well and truly over. 90' - GOAL! It's five! Abdoun races clear and tees up Reid who arrives on cue to seal victory with a thumping finish into the roof of the net. TALKING POINT: Is Big Sam under pressure? With no Premier League win in seven and a lack of options due to injuries and suspension, the Hammers look like a side devoid of belief. This heavy cup defeat will have added to the discontent in the stands and with Man City next up in the Capital One Cup and a trip to Ole Gunnar Solsjkaer's Cardiff at the weekend, you could be forgiven for wondering where their next win will come.
This was West Ham's biggest Third Round defeat in the FA Cup since 1926 (5-0 v Tottenham). (Opta Joe)
Nottingham Forest have now progressed from three of their last six FA Cup ties against teams from the Premier League. (Opta Joe)
West Ham have conceded 16 goals in their last five matches.
West Ham United have failed to keep a clean sheet in any of their last six games in all competitions.
MAN OF THE MATCH
Jamie Paterson (Nottingham Forest): Three goals and an assist. The midfielder was a constant menace and capped a day to remember with the match ball.
PLAYER RATINGS -
NOTTINGHAM FOREST: Darlow 7, Jara 7, Lascelles 7, Hobbs 7, Lichaj 7, Moussi 7, Lansbury 7, Reid 7, Abdoun 8, Paterson 9, Halford 7. Subs: Harding 6, Cox 6, Derbyshire 5.
WEST HAM: Adrian 7, Jarvis 5, Maiga 5, Morrison 5, Diarra 5, Downing 5, Potts 5, Whitehead 5, Lletget 5, Driver 5, Moncur 5. Subs: Fanimo 5, Burke 6, Turgott 6.