PythonAdministration

Administrating a Python installation boils down to the following questions

• How do I install, upgrade or uninstall Python on a system
• How do I maintain several versions of Python in parallel on the same system
• How do I install, upgrade or uninstall Python modules, possibly for only a specific version of Python

This page tries to answer these questions, first in a general section that explains things that are the same for all system types, then in more specific chapters that cover the peculiarities of the various system types.

Managing Python packages and modules

References

• Python Package Index (PyPi)
• A useful overview of Python packaging and installation tools

How Python locates packages and modules

Python locates packages and modules along a search path. This is a list of directories stored at runtime in the variable sys.path. When it is started, the Python runtime initializes this variable with the following values:

• The current directory (".")
• The content of the environment variable PYTHONPATH. This has the same syntax as the shell variable PATH.
• 0-n installation-dependent default paths. Usually this includes what is commonly called the site-packages folder - see the next section for details.

A program that knows what it is doing can change the content of sys.path to influence where modules are searched for.

site-packages

A Python installation usually includes a so-called site-packages folder. The actual folder name may be different.

The site-packages folder is a location where Python packages and modules can be installed so that when Python programs execute they can easily find their dependencies.

Refresh: The section "How to define modules" on the wiki page LearningPython explains the difference between modules and packages and how they are represented in the filesystem.

pip

pip is the Python package manager. pip manages the site-packages folder. pip gets its packages from a package repository called the "Python Package Index", which is commonly abbreviated to PyPI. The URL is https://pypi.org/.

Modern Python versions (>= 3.4) include pip by default, for older Python versions pip must be manually installed. This section from the "Python Packaging User Guide" has the details.

Note: Packages managed by pip are not the same as packages seen by the Python interpreter at runtime, i.e. packages imported by Python code. Quoting from the Python Packaging User Guide: "It is common in the Python community to refer to a distribution using the term “package”. Using the term “distribution” is often not preferred, because it can easily be confused with a Linux distribution, or another larger software distribution like Python itself."

Install a package:

pip install foo

Install a package for the current user only:

pip install --user foo

Show the user-specific base folder:

python -m site --user-base

Uninstall a package:

pip uninstall foo

List the installed packages:

pip list

List installed but outdated packages:

pip list --outdated

Upgrade an installed but outdated package:

pip install --upgrade foo

Search online for available packages:

pip search foo

Package formats

There are several types of package formats:

• Source distribution (sdist): A distribution format (usually generated using python setup.py sdist) that provides metadata and the essential source files needed for installing by pip, or for generating a so-called "Built Distribution". A source distribution requires a build step before it can be installed.
• Built distribution: A distribution format containing files and metadata that only need to be moved to the correct location on the target system, to be installed.
  • Wheel: A "Built Distribution" format introduced by PEP 427, which is intended to replace the "Egg" format.
  • The "Egg" format is another "Built Distribution" format introduced by setuptools.
  • Binary distribution: A specific kind of "Built Distribution" that contains compiled extensions.

Packaging modules

TODO

• Keywords: distutils, setuptools, easy_install
• http://pypi.python.org/pypi/setuptools
• easy_install appears to be distributed together with setuptools
• easy_install is comparable to CPAN
• Is it EasyInstall or easy_install?
• Cheese Shop or cheeseshop refer to PyPi
• Command to install pydot: "easy_install pydotpy"

Linux/Debian

Debian provides specific packages for many important Python modules. For these modules, the Python site administration task therefore simply means install, upgrade or remove Debian packages via aptitude or another package management frontend.

TODO: Research how to install custom modules, i.e. modules that are not available as Debian packages.

macOS

System version

macOS already comes with Python pre-installed in

/usr/bin/python

This is backed by a framework located here:

/System/Library/Frameworks/Python.framework

This system Python version tends to be quite old. For instance, macOS 10.14.1 still comes with Python 2.7.10, which was released around mid-2015.

Python.org distributions

Newer versions of Python for macOS can be obtained from python.org. These are installer packages. I have not tested any of these recently.

Homebrew version

The way how Homebrew organizes Python is documented here.

Homebrew offers two formulae:

• python installs the most recent Python 3.x version. The executable is python3.
• python@2 installs the most recent Python 2.x version. The executable is python2. The formula also provides python, thus overriding the system-provided Python.

The framework for both formulae is located here: /usr/local/Frameworks/Python.framework.

site-package folders are:

/usr/local/lib/python2.7/site-packages
/usr/local/lib/python3.7/site-packages

Windows/Cygwin

TODO

• Download setuptools .egg file that matches the installed Python version; download location is the setuptools page on PyPi
• Install by running sh /path/to/egg-file
• The command easy_install should now be available in the path

Windows "pure"

TODO