Six Years of Python

Introduction

I first started learning Python in 2019 and have been using it ever since. Back then, I had picked it up for a variety of reasons. First, it was one of the hottest languages around, so I wanted to know what all of that excitement was about. Second, I was about to start my master in Data Science and being proficient in Python was pretty much a requirement. Third, during my bachelor in Physics we were taught C and C++, but a few of my classmates found them to be too clunky for the kind of data analysis scripts that we were writing for our labs and switched to Python, generating some curiosity in me and other uninitiated classmates.

I wrote a post as a retrospective on what I’ve learned in these six years: the good, the bad and the ugly of the language, at least from my perspective.

The Good

Python is one of the most readable languages that I’ve ever used. Its syntax feels closer to plain English or pseudocode than most programming languages. This means that Python is very easy to learn and makes it simpler than most languages to understand someone else’s code and start contributing.

Moreover, Python is the second most used programming language in the world, according to StackOverflow. This means that here are blog posts, documentation pages and books explaining any concepts you might come across (see here for a list of my personal favourites). Moreover, this huge community of developers has produced (and been fuelled by) an impressive array of tools and libraries.

Python is the standard for data analysis and for a good reason. Although the language itself is slow, writing Python bindings for C, C++ or Rust libraries is quite easy (at least for the standards of language bindings), so that one can write the computationally intensive parts in a fast, compiled language, and provide Python APIs that call the underlying binaries. The list of packages for data analysis that make use of this trick is very long: Numpy, Pandas, Tensorflow, PyTorch and many others.

Python is also used to set up web servers and APIs: Django, FastAPI and Flask are the most used frameworks for this area of development, both inside the Python community and among all web services in general.

Finally, Python can also be used to write GUIs. It’s not the most common application, but it can be done.

The Bad

Dynamic Typing

Python is a strongly and dynamically typed language. This means that operations on objects are performed only if they are defined, but that the types of the objects are not known until runtime. This is convenient when you’re prototyping or need dynamic programming, but makes ensuring the correctness of what you’ve written much harder when you do not need this freedom.

As a partial workaround, the PEP 484 introduced type hints. They are basically type annotations that you can add to your code that inform static code analyzers such as MyPy about the types that variables should have.

This PEP was first implemented in Python 3.5 and represented a huge step forward. It was followed by many other upgrades, but correctness in Python is still harder to ensure compared to other languages, for a few reasons.

First of all, type hints are completely optional and are not used in any way by the interpreter. This behaviour was chosen both because type hints were added just as an enhancement and because the Python developers preferred to avoid introducing backwards incompatibile changes the semantics of the language. Although they’re widely adopted, they are not mandatory, so many packages you depend on do might not use them. As a result, the code that you write that directly depends on those packages will have blind spots in the typing system, reducing the overall effectiveness of static code analysis.

Secondly, even if your code and all its dependencies were to be completely typed, Python’s type system and its core syntax still present some shortcomings.

The most notable of all, in my opinion, is the impossibility of marking a variable as read-only. This, combined with the lack of visibility modifiers, make it unnecessarily hard to ensure that we never change a variable that’s meant to be constant, especially in larger codebases.

Careful testing may help you, but testing never guarantees correctness - only the absence of errors for the cases that you’ve tried. Only static code analysis can give you stronger guarantees.

Another big shortcoming of Python’s type system is the fact that it considers everything public. There is an established convention in Python of just prefixing the attributes that should be private with an underscore and to just ignore the problem because they assume that the code will be used by “consenting adults”.

Unfortunately, having private attributes makes guaranteeing many properties about data structures impossible, because if external code is allowed to modify our objects, then it might do so in an invalid way. Even well-intentioned users of our code could make mistakes.

Given these limitations, it’s worth asking: why is Python dynamically typed in the first place?

The answer is simple: dynamically typed languages are far easier to learn and to write. Therefore, dynamic typing contributes a lot to the simplicity that makes Python appealing to beginners and to non-professional developers, such as scientists or analysts who need to write code for their analysis, but whose main expertise is not coding.

In short: dynamic (and implicit) typing makes it easier to write prototypes, to experiment and to implement even complex things. However, it later slows you down when you want to ensure that your code is reliable, correct and safe.

Performance

Python is slow - really slow. The chief reason? Being a dynamically types language, it has to determine the correct way to execute each operation at runtime, which can massively increase overhead.

Let’s have a concrete example. Take the following snippet of code:

a = 1
b = 2
c = a + b

Even to perform something as trivial as a+b, Python has to first look at the type of a (int), then lookup for the definition of addition for this type and pass b to that method. This method will then check the type of b (int) to determine that summing b and a actually makes sense. Finally, some underlying C code is called to actually perform the operation.

Overall, Python is doing a lot of runtime analysis for something that could, in principle, be determined once and for all at compile-time, thus significantly reducing computational efficiency.

Python gets partially around this problem in two ways. The first one is the aforementioned trick of writing CPU-bound operations in C and then calling them from Python. The second is using Python in areas where performance isn’t a priority.

One example are I/O bound tasks, such as serving web content. In these cases, most of the time necessary for the completion of a request is spent by packages going around the network, not by the server performing computation. Therefore, it might be worth accepting a slight performance drop (and higher server costs for CPU and memory) in order to save development time. This is one of the reasons why Python is one of the most used languages for developing web services: it’s fast to write and for them CPU performance isn’t a dealbreaker.

Another use case is services for which the ratio between development cost and execution cost are skewed towards the former - such as the code for experimental data analysis or training neural networks. In these cases, you typically run the code only a handful of times before making changes or throwing it away entirely, so development speed acquires importance at the expense of code quality.

Packaging

Python has its own package manager, pip, with a number of new and old competitors. Installing and developing packages for Python, however, can be quite cumbersome, especially in cases in which there are some pre-built binary dependencies in the dependency tree. Moreover, the lack of Cargo-like feature flags makes it hard to target specific platforms and add extra features.

Python’s package ecosystem works, but the experience is suboptimal at best compared with the JavaScript or the Rust ecosystems.

Wrap Up

Explicit, static types were first created in languages like FORTRAN and C because they were needed: being able to determine the operations needed on a variable at compile time represents a huge speed up compared to determining which operations one needs to make at runtime. Computers at the time were slow and couldn’t afford to run interpreted languages.

With time, computers became more powerful, up to the point that running a dynamically typed, interpreted language like Python was feasible in some situations. This led to the creation and adoption of various interpreted languages, like Python, JavaScript or Ruby.

However, execution speed is not the only advantage that statically typed languages have: they also give stronger correctness, robustness and safety guarantees.

As the complexity of a Python program grows, the more things you have to track in your mind and the more places for a bug or security issue there are. Projects that prioritize safety, robustness or speed should consider other languages, such as Rust.

The Future

Python is still growing in popularity and will be the de-facto standard for data analysis for many years to come. As an open source, easy to learn language with a phenomenal ecosystem, it is very especially valuable for the people who write software as part of their job, but aren’t professional developers. I’m thinking of scientists, researchers, financial analysts, traders and many others. It is also useful to write prototypes, scripts and for experimenting with things that might be thrown out after a couple of weeks.

It is not necessarily the best language, however, on which to base complex production services, for the reasons that we have seen above.

Of course, countless of complex services based on Python are being used right now and have proved themselves reliable. However, the fact that Python is a common solution to some programming problems doesn’t mean that it’s the optimal one - it might actually be faster and cheaper to use other solutions.

Do you have any comments or feedback? Write me on LinkedIn!