Every AI demo is cherry picked and I'm tired of pretending otherwise

I watched a vendor show off their new code assistant for 45 minutes last Thursday in Denver. Perfect output every time, clean refactoring, even caught a bug I didn't see. Then I asked if I could try it on our actual production repo with our messy 8 year old Python codebase. Suddenly the demo machine "had connectivity issues" and they needed to wrap up. I've tested 6 tools like this since March, and every single one falls apart the moment you throw real world junk at it. The demos always use clean examples with perfect comments and straightforward logic. Why does nobody talk about how these models choke on legacy code with 400 line functions and no tests? Has anyone else gotten a vendor to run their tool on an actual messy project instead of the pretty demo?

1 comments

1 Comment

quinn58211d ago

Ha! Told them to spin up a Docker container with our junk code, they mysteriously ghosted.