Why Software Project Estimation Is Difficult

In my experience, the following factors make software project effort/cost estimation difficult:

1. It is much easier to estimate for the happy path which leaves gaps in specifications for handling edge cases and exceptions.
2. It is common to overlook non-functional requirements like performance, security, scalability. They often become the reason a project fails at scale, even if it succeeds at first. 
3. Not cleaning up code during development.
4. Frequent changes in libraries and frameworks necessitating unplanned rework.

Every if, switch, or loop condition can introduce new paths through the code. If a function has n independent Boolean conditions, the number of possible paths is up to 2^n. If a function takes multiple parameters, the combinations of edge values increase multiplicatively. 1 input has usually at least 3 edge cases, n inputs would have at least 3^n edge case combinations. If the function depends on or alters internal state, interacts with external systems, or uses asynchronous logic, you must test for edge timing, race conditions, resource exhaustion, etc. Edge cases scale faster than linearly with added logic, often closer to exponential or combinatorial growth depending on interaction depth.

Handling error cases and edge conditions often takes much more effort than the happy path due to the need for additional logic, testing, and debugging. If edge cases are not considered early, addressing them later can introduce costly redesign efforts. Unfortunately, the number and complexity of edge cases grow as the project progresses, especially if new scenarios are discovered during development or testing. Handling one error case might introduce or expose others, creating a chain of additional considerations that were not part of the original estimate.

Neglecting regular code cleanup leads to the accumulation of technical debt, which increases the time and effort required to implement similar features later in the project. New developers joining the team may struggle to understand and contribute to the codebase, further slowing development. Additionally, the effort required to fix bugs, integrate new systems, or perform upgrades can grow exponentially over time.

APIs, libraries and frameworks are typically updated every six months, and programming languages undergo significant changes every few years. While these updates bring improvements, they can impact project timelines.

All these factors can make your initial effort/time estimation 10 times less than the actual cost in the end. A rule of thumb you can use is to estimate effort/time for the happy path and multiply that by 10 to get a realistic number.

Interpreted vs compiled programming languages

With an interpreted language like Python or PHP, if you notice a bug in the web app API logic or the UI script, you can fix it and test the change immediately. This is much faster than stopping to recompile and deploy every time, as would be required in a compiled language.

Without compiled binaries, the same source code can be deployed on multiple platforms without adjustments, as long as the runtime environment is consistent.

Many modern web frameworks (e.g., Flask, Laravel Vite) have features like "hot reloading," which automatically detect changes in code and reload the application without restarting. This is much easier to implement in interpreted environments.

While these advantages are significant, they may come at the cost of runtime performance and error detection, as compilation often catches errors early. However, for most web applications, the increased development speed and flexibility usually outweigh these trade-offs.

Music: The Cardigans - My Favorite Game