The game is now 1-1 at half-time and the victors will go on to face West Ham United in the fifth round. Liverpool reacted well to falling behind and Milner was left holding his head in his hands after scooping over from close range.
The Reds lost an eventful fourth-round tie at Old Trafford 3-2 on Sunday, having taken a first-half lead through Mohamed Salah and later equalised courtesy of the Egyptian.
The visitors suddenly looked the likeliest to win it, Henderson denying Salah on two occasions, but Fabinho felled Cavani on the edge of the box. However, Bruno Fernandes' sublime free-kick 12 minutes from time helped Manchester United progress in the competition.
For Solskjaer it was sweet, 22 years to the day after scoring the victor against Liverpool in round four.
`I get frustrated at times`: Manchester United boss Solskjaer on Cavani`s movement
Manchester United star Luke Shaw impressed fans once again after keeping up his good form with a dominant display vs Fulham . Former Manchester United striker Dimitar Berbatov brushed aside any worries about Bruno Fernandes's current drop in form.
"Anthony [Martial], Marcus, Mason, Dan [James], all our forwards they have to use the time Edinson is with us to the best effect", he said following the Liverpool win", the Norwegian head coach told reporters.
Reflecting on the match, Liverpool manager Jurgen Klopp said that his side was not at their "absolute top" and made "decisive mistakes". Disappointed to give away the goals we did, which could have been stopped.
Matip has struggled with the rigours of playing three games in a week before, and Klopp apparently didn't want to risk him in their FA Cup clash against Manchester United.
'Don't worry about us, as a group we are really together and have to solve it together.