The big Spaniard made an instant impact in the 58th minute of the third round tie when he netted little more than a minute after he came on to chants of "he scores when he wants" from a spirited Welsh crowd.
Arsenal, sensing their best chance of a first trophy since 2005 was slipping through their fingers, replied with a Lukas Podolski equaliser in the 81st.
Kieran Gibbs then sent the visitors 2-1 up with a volley two minutes later before Danny Graham grabbed an equaliser from close range in the 87th.
Collecting a long ball nodded on by Nathan Dyer, Michu muscled through the defence by lifting the ball past Per Mertesacker in a strong and smooth movement before calmly slotting home.
It was his third goal against Arsenal this season, after scoring twice at the Emirates in the Premier League last month.
Arsenal, who have not won a major trophy since the FA Cup in 2005, were dumped out of the League Cup on penalties by fourth tier side Bradford City last month.
With the North Londoners sixth in the Premier League, 18 points behind leaders Manchester United, the FA Cup had looked like their best chance of taking some silverware from a highly erratic season.
Arsenal are also still in Champions League contention, although far from favourites with a match against Bayern Munich looming next month. (Reporting by Alan Baldwin, editing by John Mehaffey)