Borussia Dortmund will have to win a two-leg play-off to reach the Champions League last 16 despite last season's runners-up beating Shakhtar Donetsk.
Finishing a point behind eighth-placed Aston Villa means the German side, who took 10th place, will face Sporting or Club Brugge in the knockout phase play-off next month.
Serhou Guirassy scored a first-half double while Ramy Bensebaini put the result beyond doubt in the 79th minute after Marlon Gomes had pulled one back for the Ukrainian side early in the second period.
This was interim boss Mike Tullberg's first Champions League game in charge of Dortmund, and Guirassy made himself a nuisance by bundling a curling effort from Karim Adeyemi past Shakhtar keeper Dmytro Riznyk for the opener.
The Guinean then latched on to a through pass from Julian Brandt and finished through the legs of the goalkeeper just before half-time for his second goal of the night and ninth in eight matches in Europe this season.
Dortmund looked to be coasting to victory, but Gregor Kobel allowed the visitors back into the game when Gomes capitalised on the goalkeeper's heavy touch to embarrass him by toeing in to halve the deficit five minutes after the restart.
Shakhtar put the Bundesliga side under considerable pressure before Bensebaini sealed the win when he received a backheeled pass from Giovanni Reyna and lifted the ball over Riznyk to restore their two-goal advantage.
It was Dortmund's first win in six games, coming in Tullberg's second match in charge since the departure of Nuri Sahin after the 2-1 defeat by Bologna last week.