Secure download of python package dependencies (#7642) · Issues · Legacy / Trac

Secure download of python package dependencies

In ooniprobe we have a set of python package dependancies that must be downloaded.

They are described inside of:

https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/requirements.txt for ooniprobe

and

https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/oonib/requirements.txt

Currently pip does not do SSL verification of downloaded packages even if the repository on which the package is stored supports SSL. See: https://github.com/pypa/pip/issues/425

Moreover not all packages that are retrieved from pypi are delivered over SSL.

Crate.io is actually doing it right, though we don't have cert validation in pip so we are back to point 0 (https://crate.io/).

With @aagbsn we came up with the following strategy for doing this:

We create a mirror of all the packages we need on ooni.tpo and make a script that downloads the packages, checks that the gpg signature of the bundle of packages is good and if that is the case it will install them with pip.

Any of these strategies should work with our setup.py script and it should be possible to install it with python setup.py install.

The GPG key that is used to sign the python .eggs should be hardcoded inside of of the setup.py script.

We make a list of the SSL enabled sources of the the python packages and download them from with request (http://docs.python-requests.org/en/latest/) + and verify the certs with certifi (http://pypi.python.org/pypi/certifi) as was suggested here: https://github.com/pypa/pip/issues/425#issuecomment-10022808
We put all the dependencies as addresses to a Tor Hidden Services and download them via torsocks python setup.py install. This way authentication and encryption is handled by Tor. (perhaps this is a feature that crate.io should look into supporting?)

Notes:

Use distutils
Check if gpg is installed, if not failover to insecure mode, but warn the user of the security issues
All must be part of the setup.py script.