Massive Failure on the Product
I’ve been working with a team of 4 devs for a year on a major product. Unfortunately, today’s failure was so massive that the product might be discontinued.
During the biggest event of the year—a campaign aimed at gaining 20k+ new users—a major backend issue prevented most people from signing up.
We ended up with only about 300 new users. The owners (we work for them, kind of a software house but focusing on one product for now, the biggest one), have already said this failure was so huge that they can’t continue the contract with us.
I'm a frontend dev and almost killed my sanity developing for weeks working 12/16 hours a day
So sad :/
More Info:
Tech Stack:
Front-End: ReactJS, Styled-Components (SC), Ant Design (AntD), React Testing Library (RTL), Playwright, and Mock Service Worker (MSW).
Back-End: Python with Flask.
Server: On-premise infrastructure using Docker. While I’m not deeply familiar with the devops setup, we had three environments: development, homologation (staging), and production. Pipelines were in place to handle testing, deployments, and other processes.
The Problem:
When some users attempted to sign up with new information, the system flagged their credentials as duplicates and failed to save their data. This issue occurred because many of these users had previously made purchases as "non-users" (guests). Their purchase data, (personal id only), had been stored in an overlooked table in the database.
When these "new users" tried to register, the system recognized that their information was already present in the database, linked to their past guest purchases. As a result, it mistakenly identified their credentials as duplicates and rejected the registration attempts.
As a front-end developer, I conducted extensive unit tests and end-to-end tests covering a variety of flows. However, I could not have foreseen the existence of this table conflict on the backend. I’m not trying to place blame on anyone because, at the end of the day, we all go down in the boat together