The 1976 film follows Travis Bickle (De Niro), a disillusioned young man who works as a night shift cab driver and who fashions himself into a vigilante after meeting a child prostitute named Iris (Foster).
Speaking at the BFI London Film Festival, the prolific filmmaker admitted to feeling "depressed" for over two years after he had finished working on the crime drama because he found he didn't know who he was anymore or where he was "going to go next".
Reflecting on the period, Scorsese said: "The film stayed with me, and I think ultimately led to two and a half years of delving into just the kind of loss of where was I going to go next? Where was I going to go?
"I tried with New York, New York, I tried many different things, but I didn't know where I was going creatively and it depressed me a great deal and I almost lost out completely."
The film, he explained, had a difficult journey to the screen because its violent nature and bloody climax landed it an X rating, something that was a no-go for studio executives at the time.
"It was received very, very, very badly by the studio," Scorsese said.
"The problem is we got an X rating, and at the time an X rating meant the film was dead. But Midnight Cowboy had just come out, they had just won the Academy Awards and UA... said tell them you want to sell it to us, tell Columbia we will buy it sight unseen with the X and we'll release it."
Even so, Scorsese still had issues with Columbia Pictures: "Meantime, they wouldn't even listen to me and practically threw me out of the room with a statement saying cut it from an X, cut it for an R or we cut it.
"I had no recourse or anything, basically it was Julia Phillips and Michael Phillips [the film's producers] who talked them through it and got it granted.
"I remember Jerry Brown, his father talked about it to the censor board, they were concerned about Jodie Foster in it... I had to trim some of the violence at the end, not all of it."
The filmmaker said he realised that what he could do with the film was to give it a "muted colour scheme" which helped to tone down the red on camera and was "one of the ways to get around the bloodbath, so to speak."
Scorsese also spoke about the making of the film's infamous "You talking to me?" scene, where De Niro's Travis Bickle repeatedly takes out his gun in front of a mirror as if he is talking to another person.
The scene is iconic in its own right, and it came about when Scorsese asked his longtime collaborator to just talk freely to the mirror as much as possible.
"Bob improvised 'you talking to me?' I asked him to talk to mirror," Scorsese said.
"And the shot of the gun came from Shame, but, primarily, I was at his feet and I'm just saying 'do it again, do it again,' and he just got into a rhythm."
Scorsese credited the effectiveness of the scene to one of the film's editors Tom Rolf: "I think it's the only scene I ever gave to an editor in rushes form and walked out of the room and said 'see if you could just pull the best parts.'
"And he was a real pro, in terms of the Hollywood editors wanted to keep the director out of the room. I said 'see if you could just wean it down for me' and I gave it to him at 9:00 in the morning, 10 in the morning and about 1:00 in the afternoon he came back and said 'let me show you something.'
"I looked at it and I said 'don't touch it', and that's his cut."
Watch: The trailer for Killers of the Flower Moon: