Python Packaging for Production
My experiences from deploying
a website and a processing pipeline
By Michael Cooper
What not to do in production
virtualenv
git checkout
No packaging
What not to do in production
Not reproducible
Hard to automate
Packages can be more than just code
Solution:
setuptools + debian
setuptools: code and data files
debian packages: dependencies, daemons, configuration, users, file permissions
setuptools
Folder structure
setup.py
from setuptools import setup, find_packages��# Setup the project�setup(� name = 'rocky-server'� , version = '0.3'� , packages = find_packages()�� , install_requires =� [ 'fastavro'� , 'gevent'� , 'flask'� , 'requests'� ]
, ...� )
Running the code:
entry_points
from setuptools import setup, find_packages��# Setup the project�setup(� ...�� , entry_points =� { 'console_scripts':� [ 'rocky-server = rocky_server.main:main'� , 'rocky-tools = rocky_server.more_code:main'� ]� }
, ...� )
Data files:
package_data
from setuptools import setup, find_packages��# Setup the project�setup(� ...
� , package_data =� { 'rocky_server' : # NOTE: package/folder name, not pip name
[ 'templates/*.html'� , 'static/favicon.ico'� , 'static/*.png'� ]� }�� , ...
)
Location of data files
import pkg_resources��favicon_file = pkg_resources.resource_filename("rocky_server", "static/favicon.ico")�print favicon_file�# Output: /usr/lib/python2.7/site-packages/rocky_server/static/favicon.ico
During development
It must be installed for pkg_resources to work.
But I don't want to uninstall & install every time.
During development
virtualenv
and
pip install -e .
Development
michaelc@mic:~/code/rocky-server$ mkvirtualenv rocky-server�New python executable in rocky-server/bin/python�Installing distribute.............done.�Installing pip...............done.
�(rocky-server)michaelc@mic:~/code/rocky-server$ pip install -e .�Obtaining file:///home/michaelc/code/proxy/rocky-server� Running setup.py egg_info for package from file:///home/michaelc/code/proxy/rocky-server
� ........snip.........
Successfully installed rocky-server
(rocky-server)michaelc@mic:~/code/rocky-server$
Test without -e before deploying!
Because you will have missed something
Debian packaging
In production: use the package manager!
Metadata about your package:
DEBIAN/control
Package: rocky-server�Version: 0.5�Section: companyX�Priority: optional�Architecture: all�Depends: python, python-fastavro, python-flask(=0.9-53), python-gevent(>=0.13.6), python-requests�Maintainer: Michael Cooper <mic159@gmail.com>�Description: A testing web server
Building a deb: layout=deb
michaelc@mic:~/code/rocky-server$ mkdir pkg-root
michaelc@mic:~/code/rocky-server$ python setup.py install --install-layout=deb --prefix=/usr --root=pkg-root
Building a deb: control file
michaelc@mic:~/code/rocky-server$ mkdir pkg-root
michaelc@mic:~/code/rocky-server$ python setup.py install --install-layout=deb --prefix=/usr --root=pkg-root
michaelc@mic:~/code/rocky-server$ cp -r DEBIAN pkg-root/
Building a deb: make .deb
michaelc@mic:~/code/rocky-server$ mkdir pkg-root
michaelc@mic:~/code/rocky-server$ python setup.py install --install-layout=deb --prefix=/usr --root=pkg-root
michaelc@mic:~/code/rocky-server$ cp -r DEBIAN pkg-root/
michaelc@mic:~/code/rocky-server$ fakeroot dpkg --build pkg-root rocky-proxy.deb
Dependencies:
Its up to you
python-pipname
pip - to - deb
pip install
--ignore-installed
--install-option="--install-layout=deb"
--install-option="--prefix=pkg-root/usr"
--no-deps
flask==0.9
* experimentation required
Or, look at "stdeb"
https://github.com/astraw/stdeb
Automation:
Use a Makefile
so you can test locally, and run on your CI server.
And make changes as needed.
Configuration files
DEBIAN/conffiles
Configuration files
Copy default files into pkg-root
DEBIAN/conffiles
If the configuration file is edited, upgrades will not revert the file
Daemons
users: DEBIAN/postinst
daemons: upstart
Users:
DEBIAN/postinst
#!/bin/sh��mkdir -p /var/lib/rocky-server��# create rocky-server group�if ! getent group rocky-server >/dev/null; then� addgroup --system rocky-server�fi��# create rocky-server user�if ! getent passwd rocky-server >/dev/null; then� adduser --system --ingroup rocky-server --home /var/lib/rocky-server \� --no-create-home --gecos "rocky-server..." \� --disabled-login rocky-server�fi�
# It should own its directories?�chown -R rocky-server:rocky-server /var/lib/rocky-server�chown rocky-server:rocky-server /etc/rocky-server/config
upstart:
/etc/init.d/rocky-server.conf
description "Rocky Server"��env USER=rocky-server�env GROUP=rocky-server
# Log stdout/stderr to /var/log/upstart/rocky-server.log�console log��script� exec start-stop-daemon --chuid $USER --group $GROUP --exec /usr/bin/rocky-server --start -- \� --port 3000�end script
Version management
Version management
What we set it to:
{date}-{buildnumber}-{git_hash}
20130426-135-dc18100
Version numbers are in:
DEBIAN/control
setup.py
What we did:
sed in Makefile (please suggest better way)
Ignore setup.py
Version management: What we did
# Makefile�GIT_COMMIT ?= HEAD�BUILD_NUMBER ?= 0�PKG_VERSION_DATE = $(shell date +%Y%m%d -u)�PKG_VERSION_GIT = $(shell git rev-list $(GIT_COMMIT) --abbrev-commit --max-count 1)�PKG_VERSION := $(PKG_VERSION_DATE)-$(BUILD_NUMBER)-$(PKG_VERSION_GIT)��$(BUILD)/DEBIAN/control: DEBIAN/control� sed s/VERSION_PLACEHOLDER/$(PKG_VERSION)/ $< > $<.out� install -D -m 0644 $<.out $@� rm $<.out
The full debian build chain
The hard way?
Questions?