Elvis Costello revives an old friend on his new album Look Now

Look Now, the bittersweet, elegiac new album from Elvis Costello and his band The Imposters, has been 20 years in the making and sees the prolific songwriter work with his heroes Burt Bacharach and Carole King.

It also sees him reintroduce a character from his 2010 album National Ransom, in the opening track Under Lime.

"It's the continuation of a story that I wrote about a travelling songwriter called Jimmie who appears in a song that I wrote about eight or nine years ago called Jimmie Standing in the Rain and I just picked up the story of this character 20 years on.

"He's on a TV show, he's not a very likeable person and he's put in the charge of a young woman. And, you know, he's a bit of a liability and given a drink he could be unpleasant."

They told a young girl with a clipboard/"Just keep him amused"/ "Whatever you do, don't tell him your name"/ "Whatever you think, don't let him drink"

While those lyrics - written two years ago - have taken on new meaning in a post #MeToo world and speak to the potential abuse of power between a jaded lecherous old man and a young female production assistant, Costello insists he is not making any judgements.

"I've known people like that, I'd hate to think I'd actually been that person but this scene is one that I've seen played out backstage in the business.

"What I try to do in the song is not make a judgement but try to see the point of view of the young woman and the man in the song, who is beaten down by life. I'm not judging him from the outside."

It's the ability to assume the voices of different characters which is perhaps key to the success of a songwriter who first burst onto the scene in 1977 with the international hit Watching The Detectives. Since then, he has recorded in the region of 31 albums - his last one in 2013, a collaboration with US hip hop band The Roots.

But patience, they say, is a virtue and Look Now is the result of two decades of writing and waiting for the right opportunity to tell the stories Costello wanted to tell.

"A couple of the songs I've had for that length of time and I just wanted to make sure that they were on a record where they had the right company. When I was making records rapidly, I sometimes had reason to think that I hadn't treated some songs kindly because I was trying to jam them into the sound we had for that record.

"A good example is a song I wrote for Roy Orbison, The Comedians. If you were to hear my version of it, which we recorded in 1984, you could barely recognise it, it was a different arrangement and melody so sometimes it's good to be patient."

The album's second track, the haunting piano-driven Don't Look At Me, is another example of Costello's ability to step out of his own shoes and into another, in this case, a women watching a man cast his gaze over her.

"She's trying to work out what that look means. Is it one of estimation? Is it one of lust? Is it one of admiration? Is it one of desire? Is it sincere? Is it somebody taking advantage of the situation?

"I try to have more than one point of view. Some people like to make songs that lead up to the chorus. I'm trying to look for the different angles that other people don't cover."

The song was co-written with Burt Bacharach, one of the 20th Century's true musical geniuses, whose canon of work with partner Hal David includes songs like The Look Of Love and I Just Don't Know What To Do With Myself; and with whom Costello worked on 1998's Painted From Memory.

"We wrote a whole album of songs about 25 years ago and in the last 10 years we've written another 25 songs."

Bacharach contributes to several songs on the record

"It's recognisably him. [It's Bacharach playing the keys]. You can hear his compositional voice in the way he plays and the way he expresses himself at the piano. He gave some ideas but the song is predominantly my music. But as we were working together, he helped me finish it.

"I hear his musical voice and I wish I could write like him. My attempts to do so are other songs that you hear."

Bacharach also lends a mournful piano riff to the heart-wrenching Photographs Can Lie, in which a woman regards an old family photo with the knowledge of her father's infidelities.

In a frame/Under glass/They'll always be together/So in love/But photographs can lie

Another musical legend makes her own mark on the album. Burnt Sugar Is So Bitter was written with Carole King and has also been sitting on the shelf for more than two decades.

"We wrote it one afternoon while I was living in Dublin and I can't explain why it hasn't been on an album except none of the records suited the sound.

"I could have done it on a number of records since then, it was too good a song to bend it out of shape to fit the sound of the album and I'm really glad I did keep it. I tried to be patient.

"Sometimes taking a step back from the page really helps. All of it is driven by the story, which is one of a divorcee trying to put back together and trying to keep her kids happy. It's a story that you hear every day. It's not a mystery story."

Around March, whilst recording the finishing touches to the album, including some of his vocals, Costello was told by his doctors that he had a small but "very aggressive" tumour which would require surgery.

He underwent surgery in May but decided to go ahead with his tour dates in June - too soon as it turned out - and a month later, he cancelled a string of European dates.

"I was immensely fortunate in that it was identified... I hadn't allowed myself enough time to regain my strength. It was nothing to do with fighting cancer as such, let's be clear about that."

Some sensationalist reporting of his health scare is a subject which clearly still angers him.

"It was a little upsetting to members of my family and friends about some of the hysterical versions of the reports, tabloids in particular love to dramatise and have no sense of responsibility."

He admits the news concentrated his mind "because there was a period of four to five weeks where I was entering into the unknown". But, he says it didn't impact on the album as it was already two-thirds recorded.

"I don't think it affected my conducting of the string section because that is what I was doing the next day. I just wanted to do it well. I wasn't going up to the microphone with some morbid idea that this was my last creative act.

"Somewhere at the back of my mind, I thought, 'I'd better make sure this is really good in case I get some bad news and have to take more time off."

Following his news, there were tributes and well wishes, one of the most notable coming from the Arctic Monkeys, who performed Costello's 1978 song Lipstick Vogue and dedicated it to him.

Costello, pictured in the late 1970s and Alex Turner

With his own brand of witty, observational songwriting, Alex Turner is one of a long line of young British songwriters who owe a debt to Costello.

"I think Alex would be horrified by 'young songwriter'," laughs Costello. "He has a considerable career of his own now.

"I remember going to see them when we first came to America and they were wonderful. He's actually been on stage and played Lipstick Vogue with [my band] The Imposters.

Costello's influence extends beyond other songwriters - his song This Year's Girl is used over the opening credits of the second series of HBO's The Deuce - about the rise of the porn industry in 1970s New York, written by The Wire creator David Simon.

"It's pretty amazing to me that someone thought to do it. That's one of the things that songs can do, they can infiltrate things that you would never have imagined.

"So quite aside from Look Now we've got this song, which is about the way we look at one another. This Year's Girl was about an idea of glamour and attraction - those are all the same things that are in this album but written with a different kind of perspective.

"I wouldn't say I'm wiser or better, I'm just different because time changes you."

Look Now is out now

• More on Elvis Costello

• BBC Music homepage

• BBC Music News LIVE