Former presidents Obama, Clinton and Bush will not attend Trump's inaugural lunch