Some may say that “the hype is real”, while others may wrinkle their nose at such a claim. So what should you believe?

One of the largest recent studies on software engineering productivity done by Stanford University, has helped out with some answers. The study consists of large samples of 100 000+ developers and billions of codelines spanning across various industries and organizational sizes, from startups to enterprise level corporations, meaning findings should be pretty representative at a general level. 

So, what does it tell us then? Is the hype real? Well, the boring, but most accurate answer is that the hype may be real for some, as it depends on multiple factors and circumstances, which we will soon dive into. 

Limitations with existing studies

Stanford aren’t the first ones to address this question, however they point at some quite crucial limitations with many existing studies done on the impact of AI on engineering productivity. Many studies equate productivity with commits, PRs and tasks, which is not necessarily a good measure on productivity. Commits may be bad, and task sizes vary. In addition, commits oftentimes are bug fixes to stuff AI coded before, with legacy code potentially creating an exponential rework burden. Other studies are for example comparing two groups, one study group using AI and one control group coding manually. However, the tasks they test in these studies are often greenfield tasks, building simple things from scratch. This is the area where AI prosper the most, but most software engineering isn’t greenfield. 

Another interesting finding is that surveys basing themselves on developers´ self-assessments usually fall short as well, as the deviation between perceived productivity and measured productivity can be huge. As Yegor Denisov-Blanch puts it, people's self-assessment is “almost as good as flipping a coin”. This leads us to what is already touched on. With introducing AI you deliver more commits, but a much larger share of those commits are rework of bad code already written by AI, even though developers may have the impression of pushing a lot of value. Hence the misjudgement. 

New code ≠ Net gain

Source: Model used in pitch deck by Yegor Denisov-Blanch, Stanford University.

Despite all rework, lack of task compatibility, and self-overestimated output, at a fully general level, AI coding still increases your productivity by 30-40%, if one simply equates productivity with delivering more code. However you more often have to go back and fix the bugs that this code introduces, something that gives you a 15-25% productivity decrease, which in turn leaves you with an average productivity gain of 15-20%, across all industries and sectors. Not bad, but not great either.

Gains depends on task complexity and project maturity

The actual magnitude of the effect is not surprisingly further dependent on the nature of a codebase and complexity of a task. The following matrix illustrates the relation between task complexity (low and high) and project maturity (greenfield and brownfield) and its measured effect on productivity gains. As the matrix shows, high complexity brownfield tasks are not the best fit for AI. But low complexity greenfield tasks can benefit a lot. In other words, a startup building a not too advanced prototype for some new product is likely to experience the greatest value. 

Source: Model used in pitch deck by Yegor Denisov-Blanch, Stanford University.

With low popularity languages AI can even decrease productivity

But it is not as simple as that. This matrix is quite similar to the previous one, but project maturity is here replaced with language popularity (low and high). What one sees is that with low popularity languages, AI doesnt really help even for low complextiy tasks. Maybe even more interesting, for complex tasks using languages with low popularity, AI can actually decrease productivity. Yegor Denisov-Blanch also goes on to show a slide illustrating how the estimated productivity gain decreases pretty sharply going from 10 000 lines of code to 100 000 lines of code, with current AI capabilities. Dropping from an approximately 50% to 15% productivity gain for larger codebases (100 000 lines).

The summarize, the worst fit for AI would be mature projects with large codebases using low popularity languages. 

Source: Model used in pitch deck by Yegor Denisov-Blanch, Stanford University.

Key takeaways and worthy considerations

Some obvious questions to ask yourself before leveraging AI in coding may therefore be:

• What is the task complexity? (low vs high)

• How mature is my project? (greenfield vs brownfield)

• How large is my codebase? (small vs large)

• What is the popularity of the language I am using? (low vs high)

Having a critical mind, not putting blind faith in AI, while being ready to rewrite and modify any output, may give you the best results. 

This study, alongside others, still shows that AI does increase developer productivity, and you should probably use AI in most cases. But it does not always increase productivity to a large extent. 

Is it worth it if it only makes me 10% more efficient?

Maybe. But also maybe not. 

On the plus side, yeah, you may theoretically turn out 10% more efficient, and you get to familiarize with a technology that is poised to stay. But the downside is also apparent.

• You spend more time on prompting and less time on writing actual code. What does that do with your longtime ability to actually critically review and deep dive into AI-generated and human written code? And what does it do with your ability to make modifications and manually write great code yourself? Will developers, and especially junior ones, turn into “brain-dead” AI-servants?
  
  

• How fulfilling for an individual and good for a company's culture is it to write most code in AI? Yes, many developers today are fascinated by the technology and also feel empowered by it, but how will they feel when the hype has settled and routines have set in? If the sentiment should turn, is it worth sacrificing employees' well-being for a 10% efficiency gain? How does that affect their longterm productivity if it gradually lets them end up feeling miserable with their tasks?
  
  

• It requires tokens and subscriptions to generate the code. And maybe more tokens to maintain and refactor/rework bad code. Is it ok to spend 200 USD/month on AI-coding tools if your efficiency gain is 5%? Maybe in countries like the US or Germany, but what about Bangladesh and The Philippines? Does it make sense there? And what if the cost spikes in the coming years?
  
  

• What is the long term maintenance risk of writing more code? Will you spend more time reviewing? And is it really desirable with no human overview on what is produced? Sure, promising review tools for this bottleneck are popping up, but to what degree will they actually help you? For many tools, their long term effect is yet to be proved. 
  
  

• And what if you are handling brownfield complex code, and the gained net efficiency drops below 0, even though you feel productive? It may be hard to put an efficiency gain on the code you are working on, in particular if it is marginal, so there is always a risk that your experienced productivity is plain wrong. 

Of course, the improvements of AI-coding tools won’t stop today. With more data and better algorithms they are very likely to keep getting better. But in the noise of the hype, it is important to not become blinded, and remember that most technologies, maybe also this one, will to some degree follow Gartner's hype cycle. That said. Be curious. Be critical. Be context-aware.

Main source: https://www.youtube.com/watch?v=tbDDYKRFjhk