And not forgetting former X Factor favourite Cheryl, who has been hotly tipped to return after having her son Bear a year ago.
Host Dermot O'Leary said in recent months the show's format may be getting tweaked again: "We had a lot of twists in the last series but this time we're going back to basics".
After the call, he told The Sun: "My contract is watertight".
The Irish boy band manager, who has reportedly contacted Cowell to discuss his future on the show, said that his contract was "watertight".
"Louis and Sharon are both contracted to return this year, but Simon and the producers have lost faith".
Louise has sat on the judging panel nearly every year since its inauguration back in 2004, except when Nick Grimshaw replaced him 2015. "Louis has had a great run, but he's advertising hearing aids for God's sake", they continued.
Salah outlines two conditions to join Real Madrid in £160m deal
They only got the penalty and then they thought. somebody has to shoot. "It's very exciting. It's the quarter-finals of the Champions League and I've never played at this stage before.
"Sharon did say she was surprised she hadn't been given any filming dates".
The source also revealed that Sharon forgetting her contestants' names a year ago and Louis coming clean about a hearing problem in a recent Specsavers ad for hearing aids has raised questions over their suitability for the show.
"Nicole was a passionate judge but has no contract and won't be back this year in order to bring in fresh blood", the source added. She mentored her acts.
Scherzinger's spot on the show was already been in question, with alum Cheryl Cole being rumored to return to the series after leaving it behind to concentrate on being a motherhood.
NICOLE Scherzinger has broken her silence after being dropped from The X Factor.
Louis, who is paid £1 million per series, and Sharon, who is paid a rumoured £1.7million per series, are said to be consulting lawyers.
Sharon Osbourne was left "exhausted" previous year following her weekly flights from London to LA to fulfill her TV commitments.