New to Gold bullion

As someone who only has silver bars, mainly cause I use it as a sort of “savings” that it’s not as easy to spend as the money in my accounts, when is honestly the time to start buying gold? I know people say “the time is always now” but with gold being at an all time high, is it not reasonable to expect it to go back down soon? And lose money pretty quickly.