AI-driven development 101
Production AI driven development is achievable. Many companies in the market are doing it at a much bigger scale than what most of us is working on already.
I’ve spent the last ~4 months reading about product scalability, reliability and testing new engineering AI workflows. At first I was skeptical of relying on AI too much for unsupervised work. Now I feel much more comfortable and overall output quality has increased over time. Hopefully this helps making you comfortable as well.
AIs are great hammers. This makes it very easy to use them in the wrong way and achieve suboptimal results. It happens due to user’s lack of knowledge in the topic, lack of effort, poor understanding of AIs themselves. Some are struggling with making an MVP work while others are re-building critical systems in a short time.
How do we get from vibecoder ai slop output to reliable, production grade code? There are many articles on this topic. OpenAI has very interesting ones, together with Spotify, Anthropic to name a few. This is a summary of general guidance you can find online and my personal experience working on AI driven workflows.
Problem definition and boundaries
This is for you and your AI. If you have to rely on someone, it’s better to be as precise as possible and clear any doubts before starting work. This includes: architectural decisions, functional requirements, libraries to use, spec definitions, deployment configuration, testing coverage and so on. This work can be AI assisted and I would strongly recommend to have familiarity with the topics. It helps shape the problem in the correct way. I personally found uncomfortable using technology I have not used before and would first read about it to have a better understanding.
Problem break down
Once problem is well defined, general consensus is to break it down to individual executable tasks. Reasons why you should do this vs. one-shotting a plan:
- You have a well defined view of all the tasks required and current development status
- Each task can be worked on using a new session. Fresh context makes task execution much more reliable
- Tasks can be worked on independently at the same time in different sessions(given they do not overlap)
In this step you can also mark which tasks are higher priority, blocked by other tasks, require human review and so on. An interesting property of planning work using this method is that it naturally unfolds into a sequence of executable steps, making it easy for you and your agent to understand what is the next executable task.
Execution
You have a well defined problem and all the tasks it is made of. We now want to ensure that execution is done properly and reliably, following our guidelines and boundaries we imposed. To achieve this, we can have a WORKFLOW.md file or similar that contains execution-specific guidance:
- How to pick a task from tour task list
- How to work on it: separate worktree, ensure there is enough information to execute, reference to guidelines and specs defined at the beginning, mark the status as in progress, give the agent access to the right tools (mcps, apis etc)
- How to evaluate the work done: this can be done in many ways, from manual checks for granular control to adversarial agents checking for testing coverage, vulnerabilities, CI passes.
Each task will refer to the same workflow (or different ones if you might need granular tool access), follow a standard implementation flow and have a more consistent evaluation process.
Once this setup is complete, you can start automating and extend it. Examples:
- Workflow execution can be fully automated. You can have a script that automatically looks at the current task list, monitors which tasks can be worked on, assigns them to codex sessions and monitors them. Note that we are not asking a session to do this as this process is deterministic and a program can execute this loop reliably.
- We can apply the same concept of task execution to task creation: a set of guidelines that ensures the task has enough information to be actionable, scope is clear, and is labeled correctly. This creates a standard for task inclusion and potentially allow non technical people to work on a problem without making engineers upset.
Working example: horse dating app
Let’s say we want to build a dating app for horses. We start from defining the app, architecture etc. I will keep the example small to give an idea of how to apply this process.
Functional requirements
- Create and edit a horse profile: name, photos, age, breed, and bio.
- Browse other horses and choose like or pass.
- Create a match when two horses like each other.
- Let both horses see their matches.
The proposed structure is a web interface, an authenticated API, a database for profiles, decisions, and matches, and storage for photos. Shared data contracts describe how these parts communicate.
Then we break down the problem. Linear is my favourite tool for issue tracking and was pleasantly surprised to see openAI team going for a similar approach for their own orchestration framework. Example:
| Issue | Work | Blocked by |
|---|---|---|
| 20 | Return eligible profiles | 13, 14 |
| 21 | Save decisions and create mutual matches | 13 |
| 22 | Expose an authenticated decision endpoint | 14, 21 |
| 23 | Build the discovery screen using fixtures | 11, 12 |
| 24 | Connect the screen to the APIs | 20, 22, 23 |
Numbers are issue IDs. Prerequisites: 11 = contracts; 12 = app setup; 13 = database and fixtures; 14 = authentication.
The screen and backend can progress independently once their prerequisites are complete. Connecting them is a separate issue that waits for all three inputs.
This is how a full breakdown could look like for the full app:
We can now think about execution defining a workflow. Simplified example:
- Pick: select an enabled sub-issue with clear requirements, available tools, and completed blockers. Claim it and mark it In Progress.
- Work: follow the specifications in a separate worktree. Keep progress, decisions, and the next action recorded on the issue.
- Verify: run acceptance checks and applicable CI. Attach results and the tested revision, then submit for review.
- Handle problems: fix failed checks and review feedback within the issue. Pause when required inputs are missing; create separate issues for unrelated improvements.
- Complete: mark Done after acceptance and integration, making the result available to dependent issues.
You can now start by running this workflow manually, see where it fails, improve specifications and iterate until you feel comfortable automating it. I recommend looking at existing examples https://openai.com/index/open-source-codex-orchestration-symphony/ for inspiration and use your own experience to create what fits you and your team best.
Considerations
Context window size is not important. If you feel like context size is limiting you, you are doing something wrong. OpenAI has a monorepo, Codex has a 256k context window and they have been having no issues with it. I doubt your repository is bigger.
If you feel like you keep running into issues and poor executions you should consider re-evaluating your setup and understanding why this is happening. AIs are very capable and it’s unlikely that the problem you are working on is too hard for AI to solve. You just have to find the right approach.
I personally do not have an automated orchestration workflow yet. I do have a setup similar to what suggested and Codex skills for Linear issues -> PR creation and feedback -> linear issues creation. I manually pick a few tasks and execute them and now planning my own workflow using symphony as inspiration.
I do not believe non technical people can ship production apps, but a structured approach can allow them to iterate on ideas without creating major issues. Engineering work is still fundamental, with the focus switching to the most important decisions, delegating what is possible and eliminating dependencies between engineering and other teams. A product manager creating a demo can be useful for a proof of concept, but concepts skip all the difficult decisions.