Bump Python version

We are evaluating to generate Tor Browser offline documentation in our build pipeline (#34451 (closed)).

For this, we're using some of the repositories of the web team. They have a dependency on click==8.1.0, which is not compatible with Python 3.6.8, which is the Python version we're currently using.

In an IRC conversation I had this morning on #tor-www, emmapeel suggested me to stick with Python 3.8.x, because later versions have problems with i18n.

Notice that Firefox recommends a minimum version. As I wrote on #40478 (closed), 3.6.x is EOL, so we should be able to update to 3.8.x without any problems.

changed the description

We're currently using an old debian release for Linux builds, in order to link to an old glibc so that the binaries can work on old Linux distributions.

But for things like documentation that don't produce binaries, we can do the build with a more recent Debian release. For example we are building https-everywhere using bullseye.

And it looks like bullseye and buster have a lektor package we can use.

We're currently using an old debian release for Linux builds, in order to link to an old glibc so that the binaries can work on old Linux distributions.

It makes sense

And it looks like bullseye and buster have a lektor package we can use.

My idea was to use TPA's requirements.txt and install everything with pip, to make sure we have a configuration known for working.

Using the package would be a great idea if it still works for us and can make setup easier. Lektor downloads some plugins at the first build and I still have to investigate how to avoid this for an offline build. I see that Debian's version has an i18n.py file, but I don't know if it's what we use.

Anyway, while looking for information I found this:

Somebody must really like lektor here at Tor .

mentioned in issue #40478 (closed)

Okay, so far I've found that we aren't enabling optimizations on Python.

But enabling them (by adding --enable-optimizations to ./configure) has two disadvantages:

1. optimizations require a recent enough version of the compiler: for us it means including the compiler as input to Python (we compile GCC 10.3.0 at the moment, Jessie has 4.9.x)
2. optimizations increase compilation time by a lot, because they enable profile-guided-optimization. The profiling is obtained by running all tests in single thread! However Python is built once... and basically kept forever.

With a modern compiler it is possible to compile Python 3.7.x in Jessie (I've tried with 3.7.13). Probably also Python 3.8. I had that error related to the optimizations. Of course, I had tried to disable in the debug shell, but without running make clean, which was needed. I'll try again, if we decide to bump the Python version.

Another discovery is that we need libffi-dev to compile Python, otherwise it doesn't generate the _ctypes module, which we need for setuptools.

Finally, I wanted to try the Python version we use outside the containers, both to download the dependencies, and to be sure we don't have surprises (and so far it was a clever choice).

However, it can't create a Python with SSL support enabled, the reason seems to be that Jessie's OpenSSL is too old:

Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().

The OpenSSL we build should work, though (but it would increase the frequency of Python updates - possibly, at each release - for something we'd use only rarely if we need to update dependencies or in the debug shell!).

The alternatives are:

• solve lektor's problems with Python > 3.8: this is the ideal solution, because it means we can use standard containers to get it done
  • maybe updating lektor is enough, but we'd have a different configuration from the web team
• resolve dependencies with a Python 3.8 built in a more recent distribution - my current solution (that should have been temporary, just to avoid wasting hours in these Python problems)

TL;DR: this could be the start of a rabbit hole, so we've decided to package the offline docs outside tor-browser-build, at least as a start, and just download them from people.tpo.

So, this whole Python thing becomes optional, but we could do it anyway in (the near or distant) future.

The commit for 3.7.13 is ready (with optimizations, without OpenSSL). Updating it to a later version is possible, it just involves updating also the python.gpg keyring, because Python people rotate the signing role.

added Roadmap::Future label

added Task label

added Build System label

marked this issue as related to #40855 (closed)

assigned to @pierov

removed Roadmap::Future label

added Doing label

added FF115-esr label

marked this issue as related to #40410 (closed)

While the Python 3.6.x version we're using already matches mach's requirements, it lacks some modules we need to build Node (bz2; we were using Python 2, but now with Node 16 we need Python 3 ) and mach (ncurses).

At this point, since we're already updating the old configuration, it makes sense to also update to a more recent version.
The currently latest Python version (3.11.3) seems to be compatible with our Stretch containers, so I'd update to that one (and update the Python keyring in doing so).

changed title from Bump Python version to 3.8.x to Bump Python version

mentioned in merge request !738 (merged)

marked this issue as related to #40848 (closed)

added Needs Review label and removed Doing label

closed

removed Needs Review label

marked this issue as related to #40873 (closed)

marked this issue as related to #40965 (closed)

marked this issue as related to #40966 (closed)