Instructor said I may never be able to drive

So I (21F) have been learning to drive for just under a year, one lesson a week. Initially I was doing 2 hours per week but, due to financial constraints, had to limit it to 1 hour. My first driving instructor took me round a suburban neighbourhood for months without progressing me at all because he said I wasn't ready. My second instructor, in my first few lessons with him, took me out of suburbia and onto dual carriageways and around roundabouts and whatnot. It all seemed to be going great.

I will admit I am neither the most confident nor the best driver. I do make mistakes. Sometimes I go into third instead of first, sometimes I don't slow down fast enough, occasionally I stall. My instructor says my main problems are lack of spatial awareness (I drive too close to parked cars even though I feel like my positioning is good and can't tell I'm too close) and slow reactions (don't react to hazards quickly enough).

He is very laid back and jokey and frequently makes jokes about how I'm a bad driver. Last lesson, about halfway through, he said "mate, I'm not sure we're ever gonna get you driving," and not in a joking way.

It felt like such a punch in the gut. I got fired last month from my first "big" job due to being off sick for 3 days, so my self esteem has been pretty shot anyway. This was just another jab. For the rest of the lesson I could hardly pay attention and started making more mistakes, which made me so flustered.

I really want to be able to drive. Everyone I know can drive and here I am being told by my instructor that I may never drive. I didn't even think I was that bad a driver, for a learner anyway.

Should I find a new instructor? Give up? Take a break for a bit?

I really want to drive and, until now, I thought I'd do it eventually. Now I'm doubting myself massively and, between this and losing my job, feeling like a complete failure in life.

Any advice?