Terrified to leave my cushy govt job, have 2 years of business savings

I dont know what to do or when to leave my job. I work for the govt making 150k/year. I work maybe 2 hrs a day, but I am back in person at the office and it really feels like torture because i have to sit there for 8 hours. I know, first world problems. I literally sit at my desk at 8:30 asking myself what the hell i am doing, and that i shouldnt be here. I do work on my business at work, but its still torture to not have freedom. I have 2 months of holidays a year. My pension is at about 40k/year. I have 4 months of sick leave that i really should probably start using. They paid for my education. A lot of perks.

I have been building a business on the side for the past 3 years. Its mostly providing mental health services to patients. Im working on another income stream to feel a bit more stable. In 2024 the business made 300k and I saved 100k. So I have 300k saved in my business account. My partner works in the business and we draw approx 100k/ year salary. My partner quit their job and also takes care of the kids with our set up.

Im terrified of leaving the stability of my job, especially since its minimal work. At what point do I just do it? Or should I just keep doing this and just make this my life?

I go between “this is bearable, i will never find another job that pays this or is like this” to “this job pays nothing compared to what I can accomplish, and Im wasting my life away”

Has anyone been in this situation or have any insights?