Why hiring an Python AI Software Engineer is so difficult, and why its not your fault.
I support my customers with hiring for AI teams and recently assisted to recruit multiple experienced software engineers that should join the development of an already existing AI project with a sizable python codebase.
To be frank, the candidates where hilarious … but I don´t think its their fault.
I believe its the Python history and ecosystem that is to blame and for various reasons, Python is the wrong language to search for experienced software engineers.
Resulting in project delays, shortcuts left and right, and compromises on hiring decisions that jeopardize the long-term success of AI projects. Can we do better? Well, lets start with what we are looking for …
What is a Python AI Software Engineer?
I expect an experienced Python AI Software Engineer to be able to create high quality AI applications in a modern software stack using Python.
This includes writing modular, testable and maintainable code while being able to package, deploy and run it in the cloud using infrastructure as code.
Of course there are people out there that can do all of this very well and especially if one can draw from an global candidate pool, but to find such a candidate in the German speaking market can be a real headache.
From my experience, what you can find are two types of candidates.
Overconfident generalists, so people that think they have seen it all and can do it all. They have experience in writing software; often for quite some years, in varying languages and sometimes have a solid understanding how to run software in the cloud. What they lack is any understanding of AI that goes deeper than a shallow reading of blog posts and youtube tutorials. Often paired with only a fundamental capacity to write python code that touches more advanced topics like metaclasses, generics or asynchronous programming.
The other set of candidates, are Data Scientists that can write useful algorithms in Jupyter notebooks and are able to apply the latest AI libraries, but have no clue about building production-ready software that is managed through infrastructure as code. They are often highly motivated to try the newest tools and frameworks, but struggle with concepts like security, testing, monitoring or maintainability.
Why finding AI Engineers is so hard?
So whats the reason that it is hard to find people that can write good python software with a solid understanding of AI?
I think its primary Python’s history and its adoption as the “lingua franca” of AI.
I started using python around 2011 for multiple projects during my masters and having started programming in C, using python felt like a more sophisticated bash scripting. Python has of course come a long way from its early days as scripting language and has evolved into a language that can be picked up by everyone easily, with a rich ecosystem of libraries, tools and documentation.
Pythons versatility as a language and its ease of getting started, comes at a price.
It is used often as the glue to combines some popular libraries and apply that ball of mud to a small custom dataset. A python programmer can get a very long way without learning about software patterns, care about efficient implementation, think about software architecture or use and create abstractions. All while staying within a small code base that needs little maintenance; ignoring security like it is always done. This creates the illusion for developers that they “know the language” and can build everything after a very short time.
All of this has been of course exacerbated by the success of machine learning in the last years and the de facto standard deep learning libraries like TensorFlow and PyTorch that make it trivial to get started with an Python AI project.
It is fascinating how easy it has become to build an AI service by running some model from hugging face, make a few API calls to a cloud service and wrap it all up with a Rest API interface. For rapid prototyping without the need for maintainability or efficiency, then Python is perfect. You will be able to iterate quickly, adjust to new demands and go from ideas to proof of concepts, faster than with any other language.
If this is all you need, then the pool of python developers is huge and you don´t need to worry about who you get too much.
If you want to build a complex, multi-service system or platform that should serve customers, reliable and at scale, some ad-hoc python code, stitched together following a youtube tutorial, will not be enough.
What to do about it?
So, how can you find people that combine solid software development skills with a grounded understanding of AI? You don’t.
My recommendation is to keep the two separate and as far from one another as possible.
Hire two teams, a Software Engineering Team focusing on building scalable, maintainable and reliable cloud solutions using robust system languages like Java, Go or Rust, and a separate AI/Data Science Team that focuses on building the core AI components of your system in python by iterating fast over ideas and leveraging the newest libraries and tools for the job.
Define a very simple contract between the two and expect the AI Team to try to change it continuously. Accept the AI team to lack up to date documentation, tests, efficient implementation or security concepts and focus on encapsulating their work and how to limits the blast radius when things break for unknown reasons.
On your Software Team, follow industry standards for modern software development, focusing on reliability, maintainability, scalability and security. Try to make them the gatekeepers that protect the customers and ensure stability of your system.
Of course its easier to have one team that does it all without friction between them. That builds reliable software while iterating fast, and be on the forefront of technology, but in my experience you will just have troubles finding the right people for the job. Instead, having two teams with different responsibilities, using different technologies, made up of engineers with different background and mindset, will proof more feasible and sustainable.
Final words
While there are of course outstanding Python software engineers with deep AI knowledge, they are a rare find. If one of them should happen to stumble into your open door, be thankful and treat them well.
For the rest, trying to build a sustainable business, hire experienced software engineers with solid experience in system languages like Java/Go/Rust and a dedicated (smaller and more focused) AI team of python developers. Accept the friction between the two and leverage the strong sides of each of them.
– Manuel Pasieka