Robert Downey Jr. responded to rumors of an impending "Iron Man 4" in a recent interview.

Fans of the action film series will be disappointed to learn that there is no sequel to "Iron Man 3" in the works.

"There isn't one in the pipe," Downey told Variety on Sunday. "No, there's no plan for a fourth 'Iron Man.'" Downey spoke to the media outlet while he was in Toronto to promote his upcoming dramatic film "The Judge".

The actor, who plays Tony Stark in the "Iron Man" and "Avengers" films, signed with Marvel and Disney for two more "Avengers" sequels, which open in theaters in summer 2015 and 2018. However, he has been hesitant to sign on for any more comic book films.

The 49-year-old laughed when asked if he would "let"somebody else play Tony Stark.

"I like that the idea is that it would be up to me, like I'm casting director for Marvel," he replied.

"Don't give it to anybody else," Downey's "Judge" co-star, Robert Duvall, quipped.

Downey also jokingly discussed the possibility of "The Judge" being a film series, as he treasured working with Duvall.

"Maybe it'll be trilogy, like that other series he was in," Downey said, referring to the popular "Godfather" films in which Duvall played consigliere Tom Hagen. "I heard two of those were good."

Confirmed sequels from the Marvel franchise include a second "Avengers" movie – "The Avengers: Age of Ultron" – which will open on May 1, 2015, and "Captain America 3," which will be released on May 6, 2016. "Avengers 3" has a suspected 2017 release date, and "Thor 3" is expected to be released in 2018.

Marvel was criticized for announcing a "Captain America 3" release date that coincided with the already-announced Warner Bros. film, "Batman v. Superman: Dawn of Justice." However, Marvel Studios President Kevin Feige confirmed in July that he has no plans of moving the May 2016 premiere.

"I continue to be all for quality entertainment for moviegoers to enjoy on weekends," he told Empire.

"If it is on the same weekend, I enjoy it slightly less. But we are doing what we've always done, which is sticking to our plan and sticking to our vision for the movies going forward and we have a very large vision that we're working on for 'Cap 3′ and for all the '3's' movies, and just because another movie plops down onto one of ours, doesn't mean we are going to alter that. Maybe we should, but we're not going to."