The first Avengers film was such a success, Joss Whedon originally didn’t want to come back to direct a sequel. How could you top a movie that became one of the biggest movies of all-time? Eventually Marvel and a lot of money were able to convince Whedon to return, but after the film was released in theaters, he quickly regretted that decision. In a recent conversation with his Avengers star Mark Ruffalo, Whedon admits that he felt like a “miserable failure” after the film.

Whedon says the the process of making Avengers: Age of Ultron left him “beaten down” and “exhausted” blaming some of that on “conflicting with Marvel” and a lot of it on doubts about his own work.

I am very proud of it. The things about it that are wrong frustrate me enormously, and I had probably more of those than I had on the other movies I made. But I also got to make, for the second time, an absurdly personal movie that talked about how I felt about humanity, and what it means, in very esoteric and bizarre ways, for hundreds of billions of dollars. The fact that Marvel gave me that opportunity is so bonkers, and so beautiful, and the fact that I come off of it feeling like a miserable failure, is also bonkers, but not in a cute way.

If you read our report from the Age of Ultron set, it shouldn’t be entirely surprising. Here’s an excerpt from the opening:

Joss Whedon is tired.

It’s just about halfway through the shoot for Avengers: Age of Ultron in London, and Whedon doesn’t as much sit down for our interview as he does collapse into a chair. It’s a bright and sunny day in London, but he looks like he hasn’t seen the sunshine in weeks. He’s completely wiped out — “raggedy” as he puts it — by his schedule, which he describes thusly: “I do this, I go home, I rewrite, I go to sleep. I do this, I go home, I rewrite, I go to sleep.”

It’s also not entirely surprising that Whedon clashed creatively with Marvel. The director has previously said his relationship with the company was “really, really unpleasant” during filming and they frequently debated over which scenes should be cut from the film. At one point, Marvel demanded a scene, that Whedon wanted to cut, stay in the film and Whedon says of his response to them: “I was so beaten down at that point I was like, ‘Sure, okay.’”

Though the film still stands at a 75% on Rotten Tomatoes, there’s no doubt that audiences and critics didn’t respond to the film as well as they did to the original Avengers movie. The backlash was so intense, Whedon deleted his popular Twitter account.

Whedon reiterated that he’ll “never” make a movie like that again, but has finished the script for his next project, which hopefully doesn’t leave him feeling like a failure.

More From ScreenCrush