Developer manual

This page contains information for jdwebsite developers.

Installation

Installation is easy on a Linux-like operating system.
For Windows and Mac, it requires some manual changes or a Linux virtual machine. Instructions are given below.

Quick Install (Linux)

$ ./clean_env.sh
$ ./build_env.sh
$ source ./env/bin/activate
$ python create_local_settings.py
$ python website/manage.py createdb
$ python website/manage.py loaddata demo_data #Optional, creates admin/admin
$ python website/manage.py runserver

Linux

Installation of the full project, and running a test server, can be done in a few minutes on any Linux machine. Just follow the steps under 'detailed instructions'. There is no need for manual configuration.

System-Level Dependencies
Depending on your Linux distribution, you may need to install some system-level development packages. A list of these dependencies for Ubuntu (15.04) is given below,

python-virtualenv, python3-dev, libldap2-dev, libsasl2-dev, libxml2-dev, libxslt1-dev, zlib1g-dev, libjpeg-dev, libfreetype6-dev

Install these using the command,

$ sudo apt-get install python-virtualenv python3-dev libldap2-dev libsasl2-dev libxml2-dev libxslt1-dev zlib1g-dev libjpeg-dev libfreetype6-dev

Get the code
Create a new clone of the project,

$ git clone https://github.com/jonge-democraten/website.git

Virtual environment
Delete an existing virtual environment,

$ ./clean_env.sh

Create a virtual environment in ./env/ and install all dependencies,

$ ./build_env.sh

Activate the virtual environment,

$ source ./env/bin/activate

From now on, everything you do within the project should be from a shell with activated virtual environment.

Configure Django settings
Generate a Django SECRET_KEY and a local_settings.py,

(env) $ python create_local_settings.py

Create a database
You have to create an initial database and a root user and password for the database,

(env) $ python website/manage.py createdb

Run a test server
You can run a local test server with the command,

(env) $ python website/manage.py runserver

and visit http://127.0.0.1:8000 in your browser to view the web interface.

Hostnames

Add the following line to your /etc/hosts file to enable the subdomains,

127.0.0.1 jd.local lh.jd.local ams.jd.local

Windows

Option 1: Native

Install Python 3 and Git.
Get the code. Open a Windows command prompt and clone the website repo,

git clone https://github.com/jonge-democraten/website.git

Enter the website directory, create a virtualenv, activate it, and upgrade pip,

python -m venv env
env\Scripts\activate.bat
python -m pip install -U pip

Download the pyldap Python wheel for your Python version from http://www.lfd.uci.edu/~gohlke/pythonlibs/#pyldap Install this wheel,

pip install pyldap-<some specific version info here>.whl

Install other requirements,

pip install -r requirements.txt

Edit env\Lib\site-packages\rq\worker.py and replace def kill_horse(self, sig=signal.SIGKILL): with def kill_horse(self, sig=signal.SIGTERM):.
This is required because RQ does not support Windows. We don't need it during development. This change prevents errors.

Create local settings and initialise the database,

python create_local_settings.py
python website\manage.py migrate

Load demo data and run a development server,

python website\manage.py loaddata demo_data
python website\manage.py runserver

Visit http://127.0.0.1:8000 to view the website.

Option 2: Linux Virtual Machine

For Windows and Mac users, it is advised to install a Linux virtual machine and use this to install the project.

  • Follow the instructions and install Ubuntu on VirtualBox.
  • Follow the instructions above for installation on Linux.

Workflow

  • New features are developed on a separate feature branch.
  • The feature branch is merged with master if the feature is finished.
  • This allows to work independently on different features and still share code.
  • Push feature branch commits often to communicate what you are working on.
  • Read more about this workflow here.

Testing

All application logic code is to be unit tested. Unit tests are ideally created before development of functionality. It supports development and documents, in real code instead of text, what classes and functions are supposed to do. Feature branches are only merged if unit tests are written and all pass.
Higher level user interface actions are tested manually.

Unit tests

  • The project uses the Django unit test framework to create unit tests.
  • This framework is based on the Python unittest module.
  • Tests are defined in tests.py of the application directory.

Run the unit tests,

(env) $ python website/manage.py test website.core website.jdpages

Automated testing

Travis is used to automatically install the environment and run tests on changes in the project.
The file .travis.yml contains the Travis commands to install and test the project.

Performance

Locust can be used for load testing. It simulates users and provides a web UI to monitor the test. Easy to install and configure. Read the locust documentation.


Logging

The Python logging module is used for logging.

Example

To add log statements, simply add the following at the top of your Python file,

import logging
logger = logging.getLogger(__name__)

and add a new log statement anywhere in this file by,

logger.debug('debug log statement')
logger.warning('warning message')
logger.error('error message')

Output

The log statements include log level, application, class, function and line number of the log statement,

[2014-12-19 22:39:11] ERROR [website.tests::test_logfile() (23)]: Cannot find anything here.

Five log levels are available: debug(), info(), warning(), error() and critical().

The log statements for the applications are written to the console, if DEBUG=True, and always to debug.log and error.log. Django errors can be found in django.log.

Configuration

Logging is configured in the Django settings.py LOGGING variables. Information about configuration can be found here. New applications have to be added before logging becomes active for those applications.

Email error notifications

All admins, as defined in the ADMINS setting, will receive email notifications of all ERROR level log messages.

Confidential information

Confidential information should not be logged. During initial development, logging of confidential information is allowed if marked with a CONF tag,

logger.debug('CONF ' + member.DNA)

These will be removed before deployment.


Documentation

  • Documentation can be found in the /docs/ directory.
  • Docs are writtin in plain text with Markdown formatting.
  • MkDocs can be used to convert the doc files into nice looking html.
  • readthedocs.org hosts a html version of the documentation, generated from the markdown files.

MkDocs

You can use MkDocs to preview the documentation in the readthedocs template.
Install mkdocs using pip,

$ pip install mkdocs

run a preview server in the project directory,

$ mkdocs serve

and visit 127.0.0.1:8000 to preview the generated documentation.

Code documentation

  • Add comments to code that is not self-explanatory.
  • Use python docstrings to describe files, classes and functions.
  • Add docstrings to files, classes and functions if useful.

Example

"""
File description.
"""

class ExampleClass(Example):
    """ Class description. """

    def example_function(self):
        """
        Function description 
        on multiple lines.
        """

Code standards

  • The default Python and Django code style is used.
  • Write code as simple as possible and focus on readability.
  • Write code for others to understand and read.

"Always code as if the person who ends up maintaining your code is a violent psychopath who knows where you live. " - source

Code check

  • Flake8 is a Python tool to check code style.
  • It runs automatically on Travis after each commit.
  • You can find the Flake8 output in the latest Travis build log.

Database

Migrations

A database migration needs to be created after database structure changes in models.py,

$ python website/manage.py makemigrations <app_label>

The generated migration file is committed together with changes in models.py.
Migrations have to be carefully managed between different branches, so keep track of other branches and prepare for a merge.


Demo data

The code base of jdwebsite contains demo data to demonstrate functionality. To create demo data, dump a fixture in /website/fixtures/,

$ ./create_demo_data.sh

This fixture may be loaded when initialising the development environment (see Installation).

Caveat

For some reason, the categories of blog posts in Blogs on non-default Sites are not exported correctly. To include this information, you have to add it by hand.


Versioning

Version numbers are of the form,

1.3.9 N.N.N major.minor.micro
  • major release for everthing worth a new number
  • minor release for all functional and model (database) changes
  • micro release for bugfixes and minor view/template modifications

Pre-releases get a suffix ,

1.3.9a1 N.N.N(a/b/rc)N alpha/beta/release-candidate

PEP0440 is used as a basis for the version scheme.