It took just 17 minutes for Cliftonville to take the lead through Donnelly.
Doherty's corner from the right was met on the head by Donnelly, who made no mistake at the back post for his first goal of the season.
Just seven minutes into the second half, Larne got themselves level through Cosgrove.
Some lovely football resulted in Graham finding the Larne captain free at the back post with a pinpoint low cross, with Cosgrove doing the rest.
Midway through the second half Larne took the lead for the first time, as Ryan dispatched a spot kick having been fouled by Addis.
It was a double whammy for the Reds, who were reduced to 10 men, as Addis was sent for an early shower following his push on Ryan that led to the penalty.
Two minutes later the Inver Reds had another, with substitute O'Neill heading home from close range from a pinpoint Graham cross once again to secure the win for Tiernan Lynch's side.
It could prove to be the Larne manager's final game in charge as he continues to be linked with the vacant St Johnstone job.