Actor Jason Isaacs has spoken up to share an interesting story about his role in the Harry Potter film franchise. While we now know him for his portrayal of Voldemort servant Lucius Malfoy, he said in a new interview that he actually wanted to play the role of the bumbling and cocky Gilderoy Lockhart. Isaacs told Digital Spy that he was “mightily pissed off” when the Harry Potter casting directors told him that he would be reading for the elder Malfoy. He decided he wouldn’t take the part, informing his agent to shut it down. His reason? He was already playing Captain Hook in 2003′s Peter Pan, and he didn’t want to play another villain in a children’s story right away.

“I said to my agent, ‘When they call, tell them no, thank you very much, I am about to go play Captain Hook,’” Isaacs said, via Kotaku. He said, ‘Just think about it over the weekend’. I said, ‘No, I don’t want to think about it–it’s fun and lovely and flattering but I am not playing two children’s villains!’” Isaacs said.

But things changed for Isaacs over that fateful weekend. He was flooded with phone calls from his family, urging him to take the part of Lucius Malfoy–for their own selfish-sounding, but well-meaning reasons.

“Over the weekend, everybody I knew called me–nieces, nephews, godchildren, and then the parents,” Isaacs said. “They all tried to persuade me to take the job–not because they cared about me but because they wanted to visit the set! So I took the job and thank god I did!”

Malfoy was a Death Eater. Though he defected and survived the events of the story, he’s looking pretty worse for wear by the end. As for the Lockhart, the role eventually went to Kenneth Branagh, and he was pretty great in the part.

Although the mainline Harry Potter novel and film series is over, the spinoff Fantastic Beasts and Where to Find Them debuted in 2016 with the first of the four sequels coming in 2018. Johnny Depp will play Gellert Grindelwald in the first sequel, while Jude Law will play Albus Dumbledore.