Spent 8 hours training a model that failed because of one wrong label

I was training a computer vision model for work and kept getting terrible accuracy scores... turned out a single image in my dataset was labeled "cat" instead of "dog" and it threw everything off. That mistake cost me a full day of debugging and retraining. Has anyone else had a tiny data error balloon into a huge time sink like that?

2 comments

2 Comments

walker.andrew3d ago

Double check every single label before you start training, it saves so much time...

jason_kelly82d ago

Yeah man, learned that one the hard way. Spent a whole week training a model only to find out half my labels were slightly off. Ended up wasting more time fixing it than if I'd just checked them upfront.