Limitations of AI in product management

In designing and developing a new product the need to organize the pieces into actionable and deliverable chunks becomes extremely important. Most on my professional career has been in this space. It doesn’t matter if you are building a house, designing a large estate garden, planning a large wedding or developing a software application, the need to properly plan is paramount. With the introduction of AI into every application, for every task in every industry, it would seem as though the need to learn how to use the fancy bloated planning software would be no more. After all, the Chads (as I call the army of ai assistants always available at our fingertips) are very good at creating everyday lists that keep our lives together. What did we even do before them? So, give them that smartly worded project.md file, integrate with your favorite product development/lifecycle tool, ask the Chads to create milestones and tickets for development and estimate how long it will take to complete and voila! Planning phase complete.

I tried this with my own project wanting to see how it did. The results were comically worse than I expected. I admit, I did this as an experiment, not expecting most of it to be useful but what I didn’t expect is how useless it actually was. Using gpt sol 5.6 high was like asking a devops Primadonna to write ux/ui requirements for a working prototype. 1 st task: create login. Ah, sorry I’m being too vague what it actually gave as a first task was Cross Ops: Configure Clerk for local, canary, production, and self-hosted deployments. Perfect first step for a product that doesn’t even have a defined UI. Over the next hour the agent created 17 milestones with 125 tickets and an estimate of 15-18 months to complete with at least 12 of those months with customer outreach activities beginning in month 3 and the dashboard development beginning somewhere in like month 12. That’s just what I got out of the chaos that was trying to decipher the backlog. After reading through all the tickets, I’m not sure what we’re building.

To step back a bit, the project is in early stages but not ground 0. We have already put a substantial amount of time into discovery and exploration, we know what needs to come next and have been trying out integrations with traditional project management and development tools like jira and linear. We have well defined mockups and architecture documents. An interesting finding in the experiment is that the agent made some wild assumptions and in some places just made up whatever it wanted without following the product platform brief at all. For example it created it’s own terminology for parts of the product and totally disregarded the existing mockups and created an entirely new interface with a new color scheme, layout and flows. Planning meetings, roadmapping, and ticket creations are not yet a thing of the past. For the seasoned consultant and PM this is a relief, jobs are safe for the time being. But, for the hoards of new developers that have found the beauty of vibe coding and confidence to believe that their idea is the next billion dollar product, this may look like a serious time saver. Feed the project plan and tickets into the agent and vibe code your way into a labyrinth with a monster of a code base chasing you further into the maze.

I see the beauty in AI and I am an adopter. With many years of experience in development processes and an appreciation of disruptive technologies, I also understand it’s limitations. I’m a natural planner and like organization but I am not opposed to change and regular rework if necessary. The way the tickets were created and spit out as if an established contract felt very waterfallish, and too rigid for my liking. I think a different approach is necessary. Most organizations can design their product lifecycle by tailoring the combination of 5 basic streams: Market, Program, Discovery, Delivery, and Hardware. All need some, some need all, none need none, however the streams are customized, there’s only one rule: have a process. For the process design of my project, I want to practice what I preach and what I’ve found helpful and valuable over the years, the most reasonable starting place is discovery.

My first thought, after reading through the storm of tickets, was to delete them all and start from scratch but decided to keep them as an archive to maybe reference later (I may end up deleting them) so I created a project and labeled it v1 archive and stuck the bunch in there. With a clean slate I can get on to creating my discovery tickets. The goal of the first discovery cycle is to have an interactive mockup as well as the basic architecture and technology decisions made to start building. With the incorporation of the Chadlings into the lifecycle, discovery takes on a new form becoming faster and more iterative – this is a good time to experiment and work through assumptions. It’s a lot more iterative than it used to be with research and prototyping more easily done upfront. The goal is to try and slice the work into small functional pieces. The product contains a workspace assistant that helps the user customize their workflows and makes recommendations. I want to be cognizant of this as I build out the project and use the assistant to help me build IT out. Output from discovery should be useful information for agent interactions.

Basic discovery tickets

1. Design and build interactive mockup.
2. Decide on platform architecture
3. Decide on basic database schema
4. Decide on basic API design
5. Create basic agent
6. Setup known services
7. Create discovery document on outcomes and decisions

Time to get to work and see how this goes.