Commit 503e741b authored by Jason Robinson's avatar Jason Robinson

Switch to pip-tools for Python dependencies

Switched to ``pip-tools`` as the recommended way to install Python dependencies and cleaned the requirements files a bit. Now all the "base" dependencies, including production deployment dependencies are locked in ``requirements.txt``. The new file ``dev-requirements.txt`` includes both the base and the extra development/testing related dependencies.

It is not mandatory to use ``pip-tools`` for running a production installation. For development it is mandatory. All dependencies should be placed (unlocked) in either ``requirements/requirements.in`` (base) or ``requirements/requirements-dev.in`` (development extras). Then execute ``./compile-requirements.sh`` to update the locked dependency files after each change to the ``.in`` files. See `pip-tools <https://github.com/jazzband/pip-tools>`_ for more information.
parent feaa2746
...@@ -8,8 +8,8 @@ services: ...@@ -8,8 +8,8 @@ services:
- redis-server - redis-server
- postgresql - postgresql
install: install:
- pip install -U pip - pip install -U pip setuptools==30.4 pip-tools
- pip install -U -r requirements/test.txt - pip-sync dev-requirements.txt
- pip freeze - pip freeze
cache: cache:
directories: directories:
......
#!/usr/bin/env bash
export CUSTOM_COMPILE_COMMAND='./compile-requirements.sh'
pip-compile requirements/requirements.in $1 $2 $3
pip-compile requirements/requirements-dev.in $1 $2 $3
# Install all locally needed requirements using this
-r requirements/requirements.txt
-r requirements/requirements-dev.txt
...@@ -6,6 +6,29 @@ Changelog ...@@ -6,6 +6,29 @@ Changelog
[unreleased] [unreleased]
------------ ------------
Python dependencies
...................
Switched to ``pip-tools`` as the recommended way to install Python dependencies and cleaned the requirements files a bit. Now all the "base" dependencies, including production deployment dependencies are locked in ``requirements.txt``. The new file ``dev-requirements.txt`` includes both the base and the extra development/testing related dependencies.
To use ``pip-tools``, first install it:
::
pip install -U pip-tools
Then install dependencies:
::
# Production environment
pip-sync
# Development environment
pip-sync dev-requirements.txt
It is not mandatory to use ``pip-tools`` for running a production installation. For development it is mandatory. All dependencies should be placed (unlocked) in either ``requirements/requirements.in`` (base) or ``requirements/requirements-dev.in`` (development extras). Then execute ``./compile-requirements.sh`` to update the locked dependency files after each change to the ``.in`` files. See `pip-tools <https://github.com/jazzband/pip-tools>`_ for more information.
Added Added
..... .....
...@@ -19,6 +42,7 @@ Changed ...@@ -19,6 +42,7 @@ Changed
* Previously previews and oEmbed's for content used to only pick up "orphan" links from the content text. This meant that if there was a Markdown or HTML link, there would be no link preview or oEmbed fetched. This has now been changed. All links found in the content will be considered for preview and oEmbed. The first link to return a preview or oEmbed will be used. * Previously previews and oEmbed's for content used to only pick up "orphan" links from the content text. This meant that if there was a Markdown or HTML link, there would be no link preview or oEmbed fetched. This has now been changed. All links found in the content will be considered for preview and oEmbed. The first link to return a preview or oEmbed will be used.
* Streams URL changes: * Streams URL changes:
* All streams will now be under `/streams/` for a cleaner URL layout. So for example `/public/` is now `/streams/public/`. * All streams will now be under `/streams/` for a cleaner URL layout. So for example `/public/` is now `/streams/public/`.
* Tag stream URL has been changed from `/streams/tags/<tag>/` to `/streams/tag/<tag>/`. This small change allows us to later map `/stream/tags/` to the tags the user is following. * Tag stream URL has been changed from `/streams/tags/<tag>/` to `/streams/tag/<tag>/`. This small change allows us to later map `/stream/tags/` to the tags the user is following.
......
...@@ -63,3 +63,8 @@ As a general rule, for Python code follow PEP8, except with a 120 character line ...@@ -63,3 +63,8 @@ As a general rule, for Python code follow PEP8, except with a 120 character line
``.editorconfig`` in the repository root. ``.editorconfig`` in the repository root.
There is an ``.eslintrc`` configuration provided as well. NOTE! This is only valid for the new Vue based frontend, not JS in ``socialhome/static``. There is an ``.eslintrc`` configuration provided as well. NOTE! This is only valid for the new Vue based frontend, not JS in ``socialhome/static``.
Python dependencies
-------------------
.. include:: includes/pip_tools.rst
...@@ -28,13 +28,10 @@ Python 3.4, 3.5 and 3.6 are officially tested against. Ensure the following are ...@@ -28,13 +28,10 @@ Python 3.4, 3.5 and 3.6 are officially tested against. Ensure the following are
The file ``requirements.apt`` contains other various dependencies. You can use the ``install_os_dependencies.sh`` script to help installing these. The file ``requirements.apt`` contains other various dependencies. You can use the ``install_os_dependencies.sh`` script to help installing these.
Install requirements Install Python dependencies
.................... ...........................
:: .. include:: includes/pip_tools.rst
pip install -U setuptools pip
pip install -U -r requirements/local.txt -r requirements/test.txt
Do NPM, Bower Do NPM, Bower
............. .............
......
We use ``pip-tools`` as the way to install Python dependencies. All the "base" dependencies, including production deployment dependencies are locked in ``requirements.txt``. The file ``dev-requirements.txt`` includes both the base and the extra development/testing related dependencies.
To use ``pip-tools``, first install it:
::
# Ensure pip and setuptools are up to date as well
# We need a slightly older setuptools due to a bug in pip-tools
pip install -U pip setuptools==30.4 pip-tools
Then install dependencies:
::
# Production environment
pip-sync
# Development environment
pip-sync dev-requirements.txt
It is not mandatory to use ``pip-tools`` for running a production installation. For development it is mandatory. All dependencies should be placed (unlocked) in either ``requirements/requirements.in`` (base) or ``requirements/requirements-dev.in`` (development extras). Then execute ``./compile-requirements.sh`` to update the locked dependency files after each change to the ``.in`` files. See `pip-tools <https://github.com/jazzband/pip-tools>`_ for more information.
We use the ``pip-tools`` command to ensure dependencies are at the correct versions.
:: ::
pip install -U -r requirements/production.txt pip-sync
...@@ -236,7 +236,16 @@ Update pip and setuptools ...@@ -236,7 +236,16 @@ Update pip and setuptools
:: ::
pip install -U pip setuptools pip install -U pip setuptools==30.4
Install pip-tools
'''''''''''''''''
``pip-tools`` is a handy tool to keep environments clean and all dependencies nicely pinned.
::
pip install -U pip-tools
Get Socialhome code Get Socialhome code
''''''''''''''''''' '''''''''''''''''''
......
#!/bin/bash
pip --version >/dev/null 2>&1 || {
echo >&2 -e "\npip is required but it's not installed."
echo >&2 -e "You can install it by running the following command:\n"
echo >&2 "wget https://bootstrap.pypa.io/get-pip.py --output-document=get-pip.py; chmod +x get-pip.py; sudo -H python3 get-pip.py"
echo >&2 -e "\n"
echo >&2 -e "\nFor more information, see pip documentation: https://pip.pypa.io/en/latest/"
exit 1;
}
virtualenv --version >/dev/null 2>&1 || {
echo >&2 -e "\nvirtualenv is required but it's not installed."
echo >&2 -e "You can install it by running the following command:\n"
echo >&2 "sudo -H pip3 install virtualenv"
echo >&2 -e "\n"
echo >&2 -e "\nFor more information, see virtualenv documentation: https://virtualenv.pypa.io/en/latest/"
exit 1;
}
if [ -z "$VIRTUAL_ENV" ]; then
echo >&2 -e "\nYou need activate a virtualenv first"
echo >&2 -e 'If you do not have a virtualenv created, run the following command to create and automatically activate a new virtualenv named "venv" on current folder:\n'
echo >&2 -e "virtualenv venv --python=\`which python3\`"
echo >&2 -e "\nTo leave/disable the currently active virtualenv, run the following command:\n"
echo >&2 "deactivate"
echo >&2 -e "\nTo activate the virtualenv again, run the following command:\n"
echo >&2 "source venv/bin/activate"
echo >&2 -e "\nFor more information, see virtualenv documentation: https://virtualenv.pypa.io/en/latest/"
echo >&2 -e "\n"
exit 1;
else
pip install -r requirements/local.txt
pip install -r requirements/test.txt
pip install -r requirements.txt
fi
# This file is here because many Platforms as a Service look for # This file is here because many Platforms as a Service look for
# requirements.txt in the root directory of a project. # requirements.txt in the root directory of a project.
-r requirements/production.txt -r requirements/requirements.txt
# Wheel 0.25+ needed to install certain packages on CPython 3.5+
# like Pillow and psycopg2
# See http://bitly.com/wheel-building-fails-CPython-35
# Verified bug on Python 3.5.1
wheel==0.29.0
# Django
django==1.10.7
# Channels
channels==1.1.6
asgi_redis==1.4.2
# Configuration
django-environ==0.4.3
django-secure==1.0.1
whitenoise==3.3.0
# Permissions
django-braces==1.11.0
# Forms
django-crispy-forms==1.6.1
django-floppyforms==1.7.0
# Models
django-model-utils==3.0.0
django-enumfields==0.9.0
django-versatileimagefield==1.7.1
# User registration, either via email or social
django-allauth==0.32.0
# Python-PostgreSQL Database Adapter
psycopg2==2.7.1
# Unicode slugification
unicode-slugify==0.1.3
django-autoslug==1.9.3
# Time zones support
pytz==2017.2
# Redis support
django-redis==4.8.0
redis>=2.10.0
# Tasks
rq==0.8.0
django-rq==0.9.5
# Generic Django utils
django-extensions==1.7.9
django-memoize==2.1.0
# Federation
#federation==0.14.1
# for quick overriding a commit version
git+https://github.com/jaywink/federation.git@bcc779e006bc0af192db08a1ff8ed245c0fbd7c9#egg=federation==0.14.1.1
# Content
#django-markdownx==2.0.21
# Own markdownx fork for some tweaks:
# - disable tab when ctrl down
# - GIF upload
git+https://github.com/jaywink/django-markdownx.git@0f7b8c01906edd5ad590bb31f04685f859d6fab1#egg=django-markdownx==2.0.21.1
CommonMark==0.7.3
bleach==2.0.0
beautifulsoup4==4.6.0
lxml==3.7.3
pyembed==1.3.3
python-opengraph==0.0.2
arrow==0.10.0
django-settings-export==1.2.1
Pillow==4.2.1
# IPython for administration
ipython
# API
djangorestframework==3.6.3
django-rest-swagger==2.1.2
# User settings
django-dynamic-preferences==1.2
# UI elements
django-tables2==1.10.0
# Search
django-haystack==2.6.1
whoosh==2.7.4
# Django URLs in JS
django-js-reverse==0.7.3
# Pro-tip: Try not to put anything here. There should be no dependency in
# production that isn't in development.
-r base.txt
# uWSGI
# ------------------------------------------------
uwsgi==2.0.15
# Static and Media Storage
# ------------------------------------------------
django-storages-redux==1.3.2
# Opbeat agent for performance monitoring
# -----------------------------------------
opbeat
# Circus
# ------
circus==0.14.0
# Local development dependencies go here codecov
-r base.txt coverage
django-debug-toolbar
django-debug-toolbar-user-panel
django-test-plus
django_coverage_plugin
django==1.10.8
factory_boy
Faker
flake8
freezegun
ipdb
pytest
pytest-cov
pytest-django
pytest-env
pytest-sugar
Sphinx Sphinx
sphinx-autobuild sphinx-autobuild
# Recommonmark master has a fix for CommonMark==0.7.3 compatibility which is not released yet
git+https://github.com/rtfd/recommonmark.git@c410abb565a3c4ad4de0ce204ab2fa2db8031cf1#egg=recommonmark==0.4.0
Werkzeug Werkzeug
django-debug-toolbar # Recommonmark master has a fix for CommonMark==0.7.3 compatibility which is not released yet
django-debug-toolbar-user-panel -e git+https://github.com/rtfd/recommonmark.git@c410abb565a3c4ad4de0ce204ab2fa2db8031cf1#egg=recommonmark==0.4.0
# improved REPL
ipdb
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile --output-file requirements/requirements-dev.txt requirements/requirements-dev.in
#
-e git+https://github.com/rtfd/recommonmark.git@c410abb565a3c4ad4de0ce204ab2fa2db8031cf1#egg=recommonmark==0.4.0
alabaster==0.7.10 # via sphinx
argh==0.26.2 # via sphinx-autobuild, watchdog
babel==2.5.0 # via sphinx
certifi==2017.7.27.1 # via requests
chardet==3.0.4 # via requests
codecov==2.0.9
commonmark==0.7.4
coverage==4.4.1
decorator==4.1.2 # via ipython, traitlets
django-coverage-plugin==1.5.0
django-debug-toolbar-user-panel==1.1.0
django-debug-toolbar==1.8
django-test-plus==1.0.18
django==1.10.8
dnspython==1.15.0 # via email-validator
docutils==0.14 # via sphinx
email-validator==1.0.2 # via faker
factory-boy==2.9.2
faker==0.8.3
flake8==3.4.1
freezegun==0.3.9
future==0.16.0 # via commonmark
idna==2.6 # via email-validator, requests
imagesize==0.7.1 # via sphinx
ipdb==0.10.3
ipython-genutils==0.2.0 # via traitlets
ipython==6.1.0 # via ipdb
jedi==0.10.2 # via ipython
jinja2==2.9.6 # via sphinx
livereload==2.5.1 # via sphinx-autobuild
markupsafe==1.0 # via jinja2
mccabe==0.6.1 # via flake8
pathtools==0.1.2 # via sphinx-autobuild, watchdog
pexpect==4.2.1 # via ipython
pickleshare==0.7.4 # via ipython
port_for==0.3.1 # via sphinx-autobuild
prompt-toolkit==1.0.15 # via ipython
ptyprocess==0.5.2 # via pexpect
py==1.4.34 # via pytest
pycodestyle==2.3.1 # via flake8
pyflakes==1.5.0 # via flake8
pygments==2.2.0 # via ipython, sphinx
pytest-cov==2.5.1
pytest-django==3.1.2
pytest-env==0.6.2
pytest-sugar==0.9.0
pytest==3.2.1
python-dateutil==2.6.1 # via faker, freezegun
pytz==2017.2 # via babel
pyyaml==3.12 # via sphinx-autobuild, watchdog
requests==2.18.4 # via codecov, sphinx
simplegeneric==0.8.1 # via ipython
six==1.10.0 # via django-coverage-plugin, faker, freezegun, livereload, prompt-toolkit, python-dateutil, sphinx, traitlets
snowballstemmer==1.2.1 # via sphinx
sphinx-autobuild==0.7.1
sphinx==1.6.3
sphinxcontrib-websupport==1.0.1 # via sphinx
sqlparse==0.2.3 # via django-debug-toolbar
termcolor==1.1.0 # via pytest-sugar
tornado==4.5.2 # via livereload, sphinx-autobuild
traitlets==4.3.2 # via ipython
urllib3==1.22 # via requests
watchdog==0.8.3 # via sphinx-autobuild
wcwidth==0.1.7 # via prompt-toolkit
werkzeug==0.12.2
arrow
asgi_redis
beautifulsoup4
bleach
channels
circus
CommonMark
django-allauth
django-autoslug
django-braces
django-crispy-forms
django-dynamic-preferences==1.2
django-enumfields
django-environ
django-extensions
django-floppyforms
django-haystack
django-js-reverse
django-memoize
django-model-utils
django-redis
django-rest-swagger
django-rq
django-secure
django-settings-export
django-tables2
django-versatileimagefield
django==1.10.8
djangorestframework
ipython
lxml
opbeat
Pillow
psycopg2
pyembed
python-opengraph
pytz
redis
rq
unicode-slugify
uwsgi
whitenoise
whoosh
# Own markdownx fork for some tweaks:
# - disable tab when ctrl down
# - GIF upload
-e git+https://github.com/jaywink/django-markdownx.git@0f7b8c01906edd5ad590bb31f04685f859d6fab1#egg=django-markdownx==2.0.21.1
-e git+https://github.com/jaywink/federation.git@bcc779e006bc0af192db08a1ff8ed245c0fbd7c9#egg=federation==0.14.1.1
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile --output-file requirements/requirements.txt requirements/requirements.in
#
-e git+https://github.com/jaywink/django-markdownx.git@0f7b8c01906edd5ad590bb31f04685f859d6fab1#egg=django-markdownx==2.0.21.1
-e git+https://github.com/jaywink/federation.git@bcc779e006bc0af192db08a1ff8ed245c0fbd7c9#egg=federation==0.14.1.1
arrow==0.10.0
asgi-redis==1.4.2
asgiref==1.1.2 # via asgi-redis, channels, daphne
attrs==17.2.0 # via automat
autobahn==17.9.1 # via daphne
automat==0.6.0 # via twisted
beautifulsoup4==4.6.0
bleach==2.0.0
certifi==2017.7.27.1 # via opbeat, requests
channels==1.1.6
chardet==3.0.4 # via requests
circus==0.14.0
click==6.7 # via rq
commonmark==0.7.4
constantly==15.1.0 # via twisted
coreapi==2.3.1 # via django-rest-swagger, openapi-codec
coreschema==0.0.4 # via coreapi
cssselect==1.0.1
daphne==1.3.0 # via channels
decorator==4.1.2 # via ipython, traitlets
defusedxml==0.5.0 # via python3-openid
dirty-validators==0.4.0
django-allauth==0.33.0
django-autoslug==1.9.3
django-braces==1.11.0
django-crispy-forms==1.6.1
django-dynamic-preferences==1.2
django-enumfields==0.9.0
django-environ==0.4.4
django-extensions==1.9.0
django-floppyforms==1.7.0
django-haystack==2.6.1
django-js-reverse==0.7.3
django-memoize==2.1.0
django-model-utils==3.0.0
django-redis==4.8.0
django-rest-swagger==2.1.2
django-rq==0.9.6
django-secure==1.0.1
django-settings-export==1.2.1
django-tables2==1.10.0
django-versatileimagefield==1.7.1
django==1.10.8
djangorestframework==3.6.4
future==0.16.0 # via commonmark
html5lib==0.999999999 # via bleach
hyperlink==17.3.1 # via twisted
idna==2.6 # via requests
incremental==17.5.0 # via twisted
iowait==0.2 # via circus
ipython-genutils==0.2.0 # via traitlets
ipython==6.1.0
isodate==0.5.4 # via python-xrd
itypes==1.1.0 # via coreapi
jedi==0.10.2 # via ipython
jinja2==2.9.6 # via coreschema
jsonschema==2.6.0
lxml==3.8.0
markdown==2.6.9
markupsafe==1.0 # via jinja2
msgpack-python==0.4.8 # via asgi-redis
oauthlib==2.0.2 # via requests-oauthlib
olefile==0.44 # via pillow
opbeat==3.5.2
openapi-codec==1.3.2 # via django-rest-swagger
persisting_theory==0.2.1 # via django-dynamic-preferences
pexpect==4.2.1 # via ipython
pickleshare==0.7.4 # via ipython
pillow==4.0.0
prompt-toolkit==1.0.15 # via ipython
psutil==5.3.0 # via circus
psycopg2==2.7.3.1
ptyprocess==0.5.2 # via pexpect
pycrypto==2.6.1
pyembed==1.3.3
pygments==2.2.0 # via ipython
python-dateutil==2.6.1 # via arrow
python-opengraph==0.0.2
python-xrd==0.1
python3-openid==3.1.0 # via django-allauth
pytz==2017.2
pyzmq==16.0.2 # via circus
redis==2.10.6
requests-oauthlib==0.8.0 # via django-allauth
requests==2.18.4 # via coreapi, django-allauth, pyembed, python-opengraph, requests-oauthlib
rq==0.8.2
simplegeneric==0.8.1 # via ipython
simplejson==3.11.1 # via django-rest-swagger
six==1.10.0 # via asgi-redis, asgiref, autobahn, automat, bleach, django-dynamic-preferences, django-environ, django-extensions, html5lib, prompt-toolkit, python-dateutil, traitlets, txaio, unicode-slugify
tornado==4.5.2 # via circus
traitlets==4.3.2 # via ipython
twisted==17.5.0 # via daphne
txaio==2.8.2 # via autobahn
unicode-slugify==0.1.3
unidecode==0.4.21 # via unicode-slugify
uritemplate==3.0.0 # via coreapi
urllib3==1.22 # via opbeat, requests
uwsgi==2.0.15
wcwidth==0.1.7 # via prompt-toolkit
webencodings==0.5.1 # via html5lib
whitenoise==3.3.0
whoosh==2.7.4
zope.interface==4.4.2 # via twisted
# Test dependencies go here.
-r base.txt
codecov
coverage
django_coverage_plugin
flake8
django-test-plus
factory_boy
freezegun
Faker
# pytest!
pytest>=2.9.2
pytest-cov
pytest-django
pytest-env
pytest-sugar
pytest-warnings
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment