Development

Installation

The instructions are tested with Fedora 30 and Bash shell. There are also draft installation instructions for other Linux distributions and Windows.

Note

The instructions install virtual environments with python3 and add the Python 3 virtual environment modules to Python user script directory. This allows for example creating own Linux user for Snippy development which has an isolated virtual environment setup from global Python modules.

In case you want different virtual environment setup, you have to modify the examples.

The virtual environments are installed under ${HOME}/.cache/snippy.

Note

The installation instructions add new software packages. Execute at your own risk.

Note

The PostgreSQL adapters used with the server are installed by compiling them with the help of pip. This require working GCC toolchain. The GCC setup and configuration is not part of the Snippy documentation.

Note

Install docker-ce to be able to run test with read container. Make sure that the user who runs the Snippy tests is able to run docker commands.

Fedora

Follow the instructions to install the project on a Fedora Linux.

# Clone the project from the GitHub.
mkdir -p ${HOME}/.cache/snippy
mkdir -p ${HOME}/.local/share/snippy
mkdir -p ${HOME}/devel/snippy && cd $_
git clone https://github.com/heilaaks/snippy.git .

# Install CPython versions.
sudo dnf install -y \
    python27 \
    python34 \
    python35 \
    python36 \
    python37 \
    python38 \
    python3-devel \
    python2-devel

# Upgrade CPython versions.
sudo dnf upgrade -y \
    python27 \
    python34 \
    python35 \
    python36 \
    python37 \
    python38 \
    python3-devel \
    python2-devel

# Install PyPy versions.
sudo dnf install -y \
    pypy2 \
    pypy3 \
    pypy2-devel \
    pypy3-devel \
    postgresql-devel

# Upgrade PyPy versions.
sudo dnf upgrade -y \
    pypy2 \
    pypy3 \
    pypy2-devel \
    pypy3-devel \
    postgresql-devel

# Below are 'generic instructions' that can be used also with other
# Linux distributions.

# Upgrade pip.
pip install --upgrade pip

# Install Python virtual environments.
pip3 install --user --upgrade \
    pipenv \
    virtualenv \
    virtualenvwrapper

# Enable virtualenvwrapper and add the Python user script directory
# to the path if needed.
vi ~/.bashrc
    # Snippy development settings.
    [[ ":$PATH:" != *"${HOME}/.local/bin"* ]] && PATH="${PATH}:${HOME}/.local/bin"
    export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
    export VIRTUALENVWRAPPER_VIRTUALENV=${HOME}/.local/bin/virtualenv
    export WORKON_HOME=${HOME}/.cache/snippy/.virtualenvs
    source virtualenvwrapper.sh
    cd ${HOME}/devel/snippy
    workon snippy-python3.7
source ~/.bashrc

# Create virtual environments.
for PYTHON in python2.7 \
              python3.4 \
              python3.5 \
              python3.6 \
              python3.7 \
              python3.8 \
              pypy \
              pypy3
do
    if which ${PYTHON} > /dev/null 2>&1; then
        printf "create snippy venv for ${PYTHON}\033[37G: "
        mkvirtualenv --python $(which ${PYTHON}) snippy-${PYTHON} > /dev/null 2>&1
        if [[ -n "${VIRTUAL_ENV}" ]]; then
            printf "\033[32mOK\033[0m\n"
        else
            printf "\e[31mNOK\033[0m\n"
        fi
        deactivate > /dev/null 2>&1
    fi
done

# Install virtual environments.
for VENV in $(lsvirtualenv -b | grep snippy-py)
do
    workon ${VENV}
    printf "deploy snippy venv ${VENV}\033[37G: "
    if [[ ${VIRTUAL_ENV} == *${VENV}* ]]; then
        make upgrade-wheel PIP_CACHE=--no-cache-dir
        make install-devel PIP_CACHE=--no-cache-dir
        printf "\033[32mOK\033[0m\n"
    else
        printf "\e[31mNOK\033[0m\n"
    fi
    deactivate > /dev/null 2>&1
done

# Run tests in each virtual environment
deactivate > /dev/null 2>&1
for VENV in $(lsvirtualenv -b | grep snippy-py)
do
    workon ${VENV}
    printf "run tests in ${VENV}\033[37G: "
    if [[ ${VIRTUAL_ENV} == *${VENV}* ]]; then
        make test > /dev/null 2>&1
        if [[ $? -eq 0 ]]; then
            printf "\033[32mOK\033[0m\n"
        else
            printf "\e[31mNOK\033[0m\n"
        fi
    else
        printf "\e[31mNOK\033[0m\n"
    fi
    deactivate > /dev/null 2>&1
done

# Example how to delete Snippy virtual environments.
deactivate > /dev/null 2>&1
for VENV in $(lsvirtualenv -b | grep snippy-py)
do
    printf "delete snippy venv ${VENV}\033[37G: "
    rmvirtualenv ${VENV} > /dev/null 2>&1
    printf "\033[32mOK\033[0m\n"
done

If there is a need to compile and build Python, follow the overal instructions below.

# Enable Sqlite and pip functionality in compiled Python by installing
# additional packages.
dnf install -y \
    libsq3-devel \
    openssl-devel \
    zlib-devel

# Compile and install Python 3.8.
cd /opt
wget https://www.python.org/ftp/python/3.8.1/Python-3.8.1.tgz
tar xzf Python-3.8.1.tgz
cd Python-3.8.1
sudo ./configure --enable-optimizations
make altinstall

Python Poetry command examples

# Poetry is not yet suitable to be used. These are just for furher # references. The main problem was that the virtual environment usage # and generated setup.py were not considered good enough. # # Poetry did not install for example the latest Python third party # packages but used cached earlier versions if they were suitable. # Also the generated setup.py file was not looking good. This would # have still required maintaining the version dependencies in multiple # places.

# Check configuration. poetry config –list

# Create Python3.7 virtual environment. poetry env use python2.7 –no-ansi poetry env use python3.4 –no-ansi poetry env use python3.5 –no-ansi poetry env use python3.6 –no-ansi poetry env use python3.7 –no-ansi poetry env use python3.8 –no-ansi poetry env use pypy –no-ansi poetry env use pypy3 –no-ansi poetry env info –no-ansi poetry env list –no-ansi poetry shell poetry install -vvv –no-ansi poetry show # Show installed packages poetry show –tree poetry show –outdated

# Remove environment poetry env remove snippy-6EC4WxPe-py3.5

# Byuild wheel poetry build –no-ansi

Ubuntu

Follow the instructions to install the project on a Ubuntu Linux.

# Clone the project from the GitHub.
mkdir -p ~/devel/snippy && cd $_
git clone https://github.com/heilaaks/snippy.git .

# Install CPython versions.
sudo add-apt-repository ppa:deadsnakes/ppa -y
sudo add-apt-repository ppa:pypy/ppa -y
sudo apt-get install -y \
    python2.7  \
    python3.4 \
    python3.5 \
    python3.6 \
    python3.7 \
    python2.7-dev \
    python3.4-dev \
    python3.5-dev \
    python3.6-dev \
    python3.7-dev \
    python3.8-dev

# Install PyPy versions.
sudo apt-get install -y \
    pypy \
    pypy3 \
    pypy-dev \
    pypy3-dev \
    libpq-dev \

# Install required Python packages.
sudo apt-get install python3-pip -y
sudo apt-get install python3-distutils -y

# Follow the 'generic instructions' for the Snippy virtual environment
# installation from the Fedora chapter.

Debian

Follow the instructions to install the project on a Debian Linux.

# Clone the project from the GitHub.
mkdir -p ~/devel/snippy && cd $_
git clone https://github.com/heilaaks/snippy.git .

# Install Python virtual environments.
sudo apt-get install python3-pip -y
sudo apt-get install python3-distutils -y
pip3 install --user \
    pipenv \
    virtualenv \
    virtualenvwrapper
export PATH=${PATH}:~/.local/bin

# Follow the 'generic instructions' for the Snippy virtual environment
# installation from the Fedora chapter.

# Install CPython versions.
mkdir -p ~/devel/compile && cd $_
apt-get install sudo -y
sudo apt-get install -y \
    zlib1g-dev
wget https://www.python.org/ftp/python/3.6.8/Python-3.6.8.tgz
tar xvf Python-3.6.8.tgz
/configure --enable-optimizations
make -j8
sudo make altinstall

RHEL

Follow the instructions to install the project on a RHEL Linux.

# Clone the project from the GitHub.
mkdir -p ~/devel/snippy && cd $_
git clone https://github.com/heilaaks/snippy.git .

# Install Python virtual environments.
yum install python-pip -y
pip install --user \
    pipenv \
    virtualenv \
    virtualenvwrapper
export PATH=${PATH}:~/.local/bin

# Follow the 'generic instructions' for the Snippy virtual environment
# installation from the Fedora chapter.

Windows

Follow the instructions to install the project on a Windows.

# Install make and Python.
choco install make python

# Clone and install manually.
#
# Note! Pytest has conftest.py import error in mapped drive in Windows.
#       This prevents running the tests in mapped drives because the
#       fixtures defined in the conftest.py are not found by pytest [1].
#
#       [1] https://github.com/pytest-dev/pytest/issues/5825
#git config --global http.proxy http://<proxy>:8080
git clone https://github.com/heilaaks/snippy.git .
python -m pip install pip setuptools wheel twine --upgrade --proxy <proxy>:8080
#python -m pip install .[devel] --proxy <proxy>:8080  # Why this does not work from Windows?
python -m pip install pipenv --proxy <proxy>:8080
pipenv shell
pipenv lock -r --dev > requirements-dev.txt
#pip install setuptools-scm --proxy <proxy>:8080  # This was not needed?
pip install -r requirements-dev.txt --proxy <proxy>:8080
pip install -r requirements.txt --proxy <proxy>:8080

Workflows

Testing

After virtual environments and the Docker CE have been installed succesfully, build the Snippy container image and run the PostgreSQL database container. The PostgreSQL database is one of the supported databases and tests are run with it.

The local PostgreSQL username and passwords are synchronized with the Travis CI PostgreSQL service. The default user is postgres and the password is an empty string.

# Compile Docker image for the 'test-docker' make target.
make docker

# Start PostgreSQL in a container.
sudo docker run -d --name postgres -e POSTGRES_PASSWORD= -p 5432:5432 -d postgres

For the Snippy development, prefer a virtual environment with the latest Python release and Python 2.7. The continuous integration will run all the tests against all supported Python versions. Most of the problems can be captured before committing by running the tests with the latest Python 3 release and with the Python 2.7.

# Work in a Python virtual environment.
workon snippy-python3.7

The Snippy continuous integration will run all tests with the default SQLite, PostgreSQL and in-memory databases with the exception of the tests with a real server or with a docker container. It is recomended to run make test-server and make test-docker manually until these tests are included also into the continuous integration tests.

# Run the default development tests. This does not include real server, real
# docker container or tests with other than SQLite database.
make test

# Run all tests against PostgreSQL.
make test-postgresql

# Run all tests against in-memory database.
make test-in-memory

# Run all tests with a real server started on the same host.
make test-server

# Run all tests with a Docker container.
make test-docker

# Run tests against all supported Python versions.
make test-tox

# Run all tests.
make test-all

# Run test coverage.
make coverage

# Open coverage report in a web browser.
file:///<home>/devel/snippy/htmlcov/index.html

# Run lint.
make lint

# Clean all generated files with the exception of SQLite database file.
make clean

# Clean only the SQLite database file.
make clean-db

# Clean all generated files including the SQLite database file.
make clean-all

# Run a test with debug logs enabled.
pytest tests/test_api_create_snippet.py -k 001 -s --snippy-logs

# Run a test with PostgreSQL database.
pytest tests/test_api_create_snippet.py -k 001 -s --snippy-db postgresql

# Run a test with in-memory database.
pytest tests/test_api_create_snippet.py -k 001 -s --snippy-db in-memory

Documentation

The documentation includes manual and automated documentation. Automated documentation is extracted from source code docstrings and from the Swagger definitions in the swagger-2.0.yml file.

# Create documents.
make docs

# Open the document in a web brower.
file:///<home>/devel/snippy/docs/build/html/development.html#documentation

Shell completions

Shell completion testing is done by manually testing the commands. Install the Snippy tool and Bash completion scripts with the below examples.

# Install and upgrade Snippy.
make install

# Install the Bash completion.
python runner export --complete bash
sudo cp snippy.bash-completion /etc/bash_completion.d/snippy.bash-completion
exec bash

API design

The Swagger editor is used to update the Snippy REST API design. Run the Swagger editor in a container and update the swagger-2.0.yml design. The Swagger version 2 is the baseline for the design because it still has wider support in open source tools than version 3. The swagger-3.0.yml file is always generated with the Swagger editor from the swagger-2.0.yml.

Snippy uses JSON schemas generated from the swagger-2.0.yml. The generated JSON schemas are used in Snippy server to validate HTTP requests and in tests to validate the Snippy server HTTP responses.

If the JSON schema filenames are changed or new files are generated, the code and tests using the JSON schema files must be updated. Note that only some of the generated files are used in the JSON schema validation. The Snippy server needs only the JSON schema files that define the HTTP methods received by the server and tests need only JSON schema files that are used to define the HTTP responses.

# Start the Swagger editor.
docker run -d -p 9000:8080 --name swagger-editor swaggerapi/swagger-editor

# Edit the swagger-2.0.yml with the Swagger editor GUI in a web browser.
http://localhost:9000

# Convert the API design to OpenAPI version 3 with the Swagger editor
# and save the changes to the swagger-3.0.yml.
Edit - Convert to OpenAPI 3

# Create JSON API schema.
make jsonschema

# Run tests with the updated JSON schema.
make test

Terms

Term

Decscription

attribute
Content attribute like brief, links or tags.

category
Content category that is one of snippet, solution or reference. This can also
refer to a field category groups or tags.

collection
Collection is a set of resources objects.

content
Content is a software development note from one of the categories. A content is
stored into a resource object. In a broader view content, resource and a note can
be interpreted to have a same meaning.

field
Same as attribute. It is recommended to use the term attribute because it
refers to a commonly used term in REST API design.

operation
Command line operation like create or delete. This term refers also to a HTTP
request in case of the REST API server. This term also refers to processing of
operation from start to an end.

operation ID
Unique operation identifier (OID) allocated for all log messages generation from a
single operation.

resource
Resource is an object that can store a single content from any of the categories.

parameter
An URL parameter that defines for example a search criteria like sall or scat for a
HTTP request.

Guidelines

Commit logs

Git commit logs must follow rules from Chris Beams with explicit change types listed below. The change types are derived from a keep a changelog and a post in Writing for Developers

  1. Add new external features.

  2. Change external behavior in existing functionality.

  3. Fix bugs. Use ‘Edit’ for typo and layout corrections.

  4. Remove external feature.

  5. Deprecat a soon-to-be removed features.

  6. Security in case of vulnerabilities.

  7. Refactor code without external changes.

  8. Edit small fixes like typo and layout fixes.

  9. Test new test cases.

The rule must be applied so that the logs are written for humans. This means that the commit log must tell the reasons and design decisions behind the change.

This rule tries also encourage a common look and feel for commit logs.

Design

REST API

The rest API follows JSON API specification version 1.1. The PATCH method works as defined in the RFC 7386 - JSON Merge Patch.

Resource attributes

Category

The category is defined when the resource is created. After this, it cannot be changed by updating it. The only way to change this attribute is to delete and create the resource again.

Data

The data attribute contains the actual content. This is a mandatory field for the snippets and solutions contents. Because the references are links, they do not store the information into the data attribute but into the links attribute.

Brief

The brief is an optional attribute. It is a one-liner that describes the content very briefly.

Description

The description is an optional attribute. This is a longer explanation that provides more details than the brief attribute.

Name

The name is an optional attribute. This field is intended to work as a short name for the content. The intention is that this attribute could be used to refer and run for example a snippet content. The field does not have restrictions on characters but it should not contain whitespaces to allow to use it as a target or a name for an action.

Groups

The groups is an optional attribute. This field can store multiple values. The desing and usage is left to user how the groups are defined. The field does not have restrictions on characters but it is recommended that a group does not contain whitespaces.

Tags

The tags is an optional attribute. This field can store multiple values. The desing and usage is left to user how the tags are defined. The field does not have restrictions on characters but it is recommended that a group does not contain whitespaces.

Links

The links is a mandatory attribute for references content. For snippets and solutions the field is optional. This field can store multiple values. The desing and usage is left to user how the fields are defined. The field is intended to store HTTP links. There is nothing that prevents storing for example file URI references.

Source

The source is an optional attribute. This field can store one value. The field is designed to contain a link to a source where the content was imported. For example the default content in snippet can set the source field to value https://github.com/heilaaks/snippy.

Versions

The versions is an optional attribute. This field can store multiple values. This field is designed to contain information related to software versions used in the content. For example commands may change between software versions. This fields tries to track version information of the content. The field accepts only key-value pairs with mathematical operators >=, <=, !=, ==, ‘>``, ‘<`` or ~. For example python>=3.7 is a valid value for the field.

Languages

The languages is an optional attribute. This field can store multiple values. The field is intended to store programming languages related to the content.

Filename

The filename is an optional attribute. This field can store one values. The field is intended to store filenames like how-to-debug-nging.md. This field can be used for example to automatically define the exported filename and format.

Created

The created timestamp is set when the resource is created and it cannot be modified. The only way to change this attribute is to delete and create the resource again.

Updated

The updated timestamp is set when the resource is updated and it cannot be modified.

Uuid

The UUID attribute is intended to be externally visible UUID attribute that is exposed outside the REST API server. The UUID is allocated always for the resource and it never changes. In order to allocate one static identifier for each resource, an UUID4 formatted ID attribute is used.

There is always a change of an UUID4 collitions if multiple servers are run but it is considered so small that it does not matter.

Digest

The digest attribute is always set by the tool based on sha256 hash algorithm. The digest is automatically updated when resource attributes are updated.

Updating

Following attributes can be freely modified by user within the limits of attribute definitions.

  • data

  • brief

  • description

  • name

  • group

  • tags

  • links

  • source

  • versions

  • languages

  • filename

Attributes that cannot be changed by user are:

  • category

  • created

  • updated

  • uuid

  • digest

Error handling

Operations will flow from the beginning to an end. There are no intermediate exists or shortcuts. The error handling must be made simple in order to keep the implementation size and testing effort in control. The target is not to try to recover all possible errors but to fail operation as soon as the first failure is detected by setting an error cause.

For example if the search category --scat option has multiple categories and one of them is faulty, the --scat option will be invalidated. This will not generate any search hits and it will minimize the database queries.

The REST API must invalidate the HTTP request if any of the attributes or parameters is incorrect. That is, the REST API must not store valid values of for attributes and silently set faulty attributes to a default values.

Testing

Security

Documentation

Docstrings

Use the Google docstring format. It is considered less verbose and more readable than the NumPy formatted docstring. In this project the intention is that a method description explains the complicated parts and the method argument is a short explanation of the method arguments. The NumPy format is better suited for complex algorithms and their parameters.

Test cases

Each test cases must have a docstring that defines the purpose of the test. The test case docstring must have a one liner brief description and empty line after the brief. The description part of the test case docstring can be freely formatted.

@staticmethod
@pytest.mark.usefixtures('default-snippets')
def test_cli_search_snippet_001(snippy, capsys):
   """Search snippets with ``--sall`` option.

   Search snippets from all resource attributes. A ``brief`` attribute
   of a single snippet produces a match.
   """

Heroku app

Use the examples below to deploy a Heroky Application.

# Install Heroku command line application.
curl https://cli-assets.heroku.com/install.sh | sh

# Run Heroku app locally.
heroku login
heroku local web -f Procfile
heroku logs -a snippy-server

# Login
https://snippy-server.herokuapp.com/api/snippets?sall=docker&limit=5

Modules

snippy.logger

Description

Logger class offers logger for each caller based on the given module name. The configuration is controlled by global settings that are inherited by every logger.

The effective log level for all the loggers created under the ‘snippy’ logger namespace is inherited from the root logger which controls the log level. This relies on that the module level logger does not set the level and it remains as NOTSET. This causes module level logger to propagate the log record to parent where it eventually reaches the snippy top level namespace that is just below the root logger.

Design

Note

This chapter describes the Snippy logging design and rules, not the Logger class behaviour.

Note

The are the logging rules that must be followed.

  1. Only OK or NOK with cause text must be printed with default settings.

  2. There must be no logs printed to user.

  3. There must be no exceptions printed to user.

  4. Exceptions logs are printed as INFO and all other logs as DEBUG.

  5. Variables printed in logs must be separated with colon.

  6. All other than error logs must be printed as lower case string.

  7. The –debug option must print logs without filters in full-length.

  8. The -vv option must print logs in lower case and one log per line.

  9. All external libraries must follow the same log format.

  10. All logs must be printed to stdout.

Overview

There are two levels of logging verbosity. All logs are printed in full length without modifications with the --debug option unless the maximum log message length for safety and security reason is exceeded. The very verbose option -vv prints limited length log messages with all lower case letters.

There are two formats for logs: text (default) and JSON. JSON logs can be enabled with the --log-json option. A JSON log has more information fields than the text formatted log. When the -vv option is used with JSON logs, it truncates log message in the same way as with the text logs.

All logs including Gunicorn server logs, are formatted to match format defined in this logger.

All logs are printed to stdout with the exception of command line parse failures that are printed to stdout.

Text logs are optimized for a local development done by for humans and JSON logs for automation and analytics.

There are no logs printed to users by default. This applies also to error logs.

Timestamps

Timestamps are in local time with text formatted logs. In case of JSON logs, the timestamp is in GMT time zone and it follows strictly the ISO8601 format. Text log timestamp is presented in millisecond granularity and JSON log in microsecond granularity.

Python 2 does not support timezone parsing. The %z directive is available only from Python 3.2 onwards. From Python 3.7 and onwards, the datetime strptime is able to parse timezone in format that includes colon delimiter in UTC offset.

>>> import datetime
>>>
>>> timestamp = '2018-02-02T02:02:02.000001+00:00'
>>>
>>> # Python 3.7 and later
>>> datetime.datetime.strptime(timestamp, '%Y-%m-%dT%H:%M:%S.%f%z')
>>>
>>> # Python 3 before 3.7
>>> timestamp = timestamp.replace('+00:00', '+0000')
>>> datetime.datetime.strptime(timestamp, '%Y-%m-%dT%H:%M:%S.%f%z')
>>>
>>> # Python 2.7
>>> timestamp = timestamp[:-6]  # Remove last '+00:00'.
>>> datetime.datetime.strptime(timestamp, '%Y-%m-%dT%H:%M:%S.%f')

Log levels

The log levels are are from Python logger but they follow severity level names from RFC 5424. There is a custom security level reserved only for security events.

Operation ID (OID)

All logs include operation ID that uniquely identifies all logs within specific operation. The operation ID must be refreshed by logger user after each operation is completed or the method must be wrapped with the @Logger.timeit decorator which takes care of the OID refreshing.

Security

There is a custom security level above critical level. This log level must be used only when there is a suspected security related event.

There is a hard maximum for log messages length for safety and security reasons. This tries to prevent extremely long log messages which may cause problems for the server.

Examples

# Variable printed at the end of log message is separated with colon.
2018-06-03 19:20:54.838 snippy[5756] [d] [b339bab5]: configured option server: true

# Variable printed in the middle of log message is separated colons and
# space from both sides. The purpose is to provide possibility to allow
# log message post processing and to parse variables from log messages.
2018-06-03 19:20:54.838 snippy[5756] [d] [b339bab5]: server ip: 127.0.0.1 :and port: 8080

logger: Logging services.

snippy.logger.getrandbits(k) → x. Generates an int with k random bits.
class snippy.logger.Logger

Global logging service.

classmethod get_logger(name='snippy.logger')

Get logger.

A custom logger adapater is returned to support a custom log level and additional logging parameters.

Parameters

name (str) – Name of the module that requests a Logger.

Returns

CustomLoggerAdapter logger to be used by caller.

Return type

obj

classmethod configure(config)

Set and update logger configuration.

The debug and very_verbose options have precedence over the quiet option. That is, either of the debug options are enabled, the quiet option does not have any effect.

Parameters

config (dict) – Logger configuration dictionary.

Examples

>>> Logger.configure({'debug': True,
>>>                   'log_json': True,
>>>                   'log_msg_max': Logger.DEFAULT_LOG_MSG_MAX,
>>>                   'quiet': False,
>>>                   'very_verbose': False})
static reset()

Reset log level to default.

static remove()

Delete all logger handlers.

classmethod refresh_oid()

Refresh operation ID (OID).

The OID is used to separate logs within one operation. The helps post-processing of the logs by allowing for example querying all the logs in failing operation.

classmethod print_stdout(message)

Print output to stdout.

Take care of nasty details like broken pipe when printing to stdout.

Parameters

message (str) – Text string to be printed to stdout.

classmethod print_status(status)

Print status information like exit cause or server running.

Print user formatted log messages unless the JSON log formating is enabled. The debug and very_verbose options have precedence over the quiet option.

If JSON logs are used, the format of the log must always be JSON. This is important for server installation where post processing of logs might be done elsewhere and incorrectly formatted logs may be discarded or cause errors.

In order to post process a JSON log, the dictionary structure must always follow the same format. Because of this, the log is pushed always as a debug level log regardless of the log level to get the formatting done.

Parameters

status (str) – Status to be printed on stdout.

static timeit(method=None, refresh_oid=False)

Time method by measuring it latency.

The operation ID (OID) is refreshed at the end.

Parameters

  • method (str) – Name of the method calling the timeit.

  • refresh_oid (bool) – Define if operation ID is refreshed or not.

Returns

Timeit wrapper function for decorators.

Return type

obj

static remove_ansi(message)

Remove all ANSI escape codes from log message.

Parameters

message (str) – Log message which ANSI escape codes are removed.

Returns

Same log message but without ANSI escape codes.

Return type

str

static debug()

Debug Logger by printing logging hierarchy.

class snippy.logger.CustomLoggerAdapter(logger, extra)

Custom logger adapter.

The logging.LoggerAdapter does not support custom log levels and therefore they need to be implemented here.

security(msg, *args, **kwargs)

Customer log level for security events.

Parameters

msg (str) – Log message as a string.

class snippy.logger.CustomFormatter(*args, **kwargs)

Custom log formatting.

format(record)

Format log record.

Text logs are optimized for a local development done by for humans and JSON logs for automation and analytics. Text logs are printed by default unless the log_json option is activated.

The debug option prints logs “as is” in full length unless the log message security limit is reached. Text logs are pretty printed with the debug option.

The very_verbose option truncates log message to try to keep one log per line for easier reading. The very verbose option prints the whole log in all lower case letters. The very verbose option is made for a local development to provide faster overview of logs compared to debug option output.

There is a maximum limitation for log message for safety and security reasons. The security maximum is tested after the very verbose option because it already truncates the log.

Gunicorn logs have special conversion for info level logs. In order to follow the Snippy logging standard, which defines the usage of debug level, Gunicorn informative logs are converted to debug level logs. Warning and error levels do not get converted because in these cases the level is considered relevant for user.

Parameters

record (obj) – Logging module LogRecord.

Returns

Log string.

Return type

str

formatTime(record, datefmt=None)

Format log timestamp.

JSON logs are printed in ISO8601 format with UTC timestamps. All other logs are printed in local time with space between date and time instead of ‘T’ because of better readability.

The ISO8601 formatted JSON timestamp is set in microseconds. It seems that the msecs field of the logging record contains mseconds as floating point number. It is assumed that the microseconds can be read by reading three significat digits after point.

Python 2 does not support timezone parsing. The %z directive is available only from Python 3.2 onwards. From Python 3.7 and onwards, the datetime strptime is able to parse timezone in format that includes colon delimiter in UTC offset.

Parameters

  • record (obj) – Logging module LogRecord.

  • datefmt (str) – Datetime format as accepted by time.strftime().

Returns

Log timestamp in string format.

Return type

str

Examples

>>> import datetime
>>>
>>> timestamp = '2018-02-02T02:02:02.000001+00:00'
>>>
>>> # Python 3.7 and later
>>> datetime.datetime.strptime(timestamp, '%Y-%m-%dT%H:%M:%S.%f%z')
>>>
>>> # Python 3 before 3.7
>>> timestamp = timestamp.replace('+00:00', '+0000')
>>> datetime.datetime.strptime(timestamp, '%Y-%m-%dT%H:%M:%S.%f%z')
>>>
>>> # Python 2.7
>>> timestamp = timestamp[:-6]  # Remove last '+00:00'.
>>> datetime.datetime.strptime(timestamp, '%Y-%m-%dT%H:%M:%S.%f')
class snippy.logger.CustomFilter(name='')

Customer log filter.

filter(record)

Filtering with dynamic operation ID (OID) setting.

Parameters

record (obj) – Logging module LogRecord.

class snippy.logger.CustomGunicornLogger

Custom logger for Gunicorn HTTP server.

setup(cfg)

Custom setup.

Disable all handlers under the ‘gunicorn’ namespace and prevent log propagation to root logger. The loggers under the ‘snippy’ namespace will take care of the log writing for Gunicorn server.

Both Gunicor error and access log categories are printed from the same namespace. In case of ‘snippy.server.gunicorn.error’, informative logs in JSON format would have this in the class name attribute which is considered to be misleading for other than error logs.

Parameters

cfg (obj) – The Gunicorn server class Config() object.

snippy.cause

Service

Cause class offers storage services for normal and error causes. The causes are stored in a list where user can get all the failues that happened for example during the operation.

All causes are operated with predefind constants for HTTP causes and short descriptions of the event.

class snippy.cause.Cause

Cause code services.

classmethod reset()

Reset cause to initial value.

classmethod push(status, message)

Append cause to list.

Message will always contain only the string till the first newline. The reason is that the message may be coming from an exception which message may contain multiple lines. In this case it is always assumed that the first line contains the actual exception message. The whole message is always printed into log.

Parameters

  • status (str) – One of the predefined HTTP status codes.

  • message (str) – Description of the cause.

Examples

>>> Cause.push(Cause.HTTP_CREATED, 'content created')
classmethod insert(status, message)

Insert cause as a first cause.

Parameters

  • status (str) – One of the predefined HTTP status codes.

  • message (str) – Description of the cause.

Examples

>>> Cause.insert(Cause.HTTP_CREATED, 'content created')
classmethod is_ok()

Test if errors were detected.

The status is considered ok in following cases:

  1. There are no errors at all.

  2. There are only accepted error codes.

  3. Content has been created and there are only 409 Conflict errors.

The last case is a special case. It is considered as a successful case when there are multiple resources imported and some of them are already created.

Returns

Define if the cause list can be considered ok.

Return type

bool

classmethod http_status()

Return the HTTP status.

classmethod json_message()

Return errors in JSON data structure.

classmethod get_message()

Return cause message.

Cause codes follow the same rules as the logs with the title or message. If there are variables within the message, the variables are separated with colon. The end user message is beautified so that if there is more than one colon, it indicates that variable is in the middle of the message. This is not considered good layout for command line interface messages.

How ever, if there is only one colon, it is used to sepatate the last part which is considered clear for user.

Because of these rules, the colon delimiters are removed only if there is more than one.

Examples

  1. cannot use empty content uuid for: delete :operation

  2. cannot find content with content uuid: 1234567

classmethod print_message()

Print cause message.

classmethod print_failure()

Print only failure message.

classmethod debug()

Debug Cause.

snippy.config

Service

Global configuration.

class snippy.config.config.Config

Global configuration object.

classmethod init(args)

Initialize global configuration.

classmethod load(source)

Load dynamic configuration from source.

classmethod reset()

Reset configuration.

classmethod get_collection(updates=None)

Get collection of resources.

Read collection of resources from the used configuration source. If resource updates are provided on top of the configured content, the updates are merged or migrated to the configuration.

Parameters

updates (obj) – Updates for the configured content in a Resource().

Returns

Configured content in a Collection().

Return type

obj

classmethod get_resource(updates)

Get resource.

Read a resource from the used configuration source. If an update is provided on top of configured content, the update is merged or migrated on top of configuration.

Parameters

updates (obj) – Resource() to be used on top of the configuration.

Returns

Updated Resource().

Return type

obj

classmethod server_schema()

Get REST API JSON validation schema.

Returns

Server API schema to validate incoming HTTP requests.

Return type

str

classmethod content_schema()

Get content attribute JSON schema.

Returns

Content schema to validate attributes for clients.

Return type

str

classmethod schema_base_uri()

Get server API schema base URI.

Returns

Path where the API schema is stored in URI format.

Return type

str

classmethod get_operation_file(collection=None)

Return filename for the operation.

Deciding filename and format with export operation:

  1. The --file option always overrides any filename and format in case of export operation.

  2. When the --file option is not used, the --format option always overrides the file format in case of export operation.

  3. If --file or --format command line options are not used, the resource filename attribute will define the filename and format in case of export operation.

  4. If none of the above conditions are met, default values are used.

If collection is provided with more than one resource, the operation file is still updated. The collection contain resources from different category than originally defined.

Parameters

collection (obj) – Resources in Collection() container.

Returns

Operation filename.

Return type

str

classmethod get_operation_files()

Return operation files.

If there is only one filename, it can be for example extracted from --defaults or --complete options. Only if there are multiple files, they are assumed to be a list of content files.

The method filters out all files which file format is not supported.

Returns

Operation filenames.

Return type

tuple

classmethod get_plugin_uri()

Return URI for the plugin.

This method reads the --file option directly as is to be used with a plugin. The URI is always the first filename in the operation file list. I case of file globs, there can be multiple source files but this not supported with the plugin URI.

Returns

URI to be used with the plugin.

Return type

str

classmethod is_supported_file_format()

Test if file format is supported.

classmethod default_content_file(category)

Return default content file.

Parameters

category (str) – User defined content category.

Returns

Filename with absolute path.

Return type

str

classmethod validate_search_context(collection, operation)

Validate content search context.

classmethod is_search_criteria()

Test if any of the search criterias were used.

static utcnow()

Get UTC time stamp in ISO8601 format.

classmethod debug()

Debug Config.

Do not print any configuration attrubutes from here. Use only the string presentation of the Config class to print attributes. This is because of security reasons.

snippy.config.source.cli

Service

Command line configuration source.

class snippy.config.source.cli.Cli(args)

Command line argument parser.

snippy.config.source.api

Service

REST API configuration source.

class snippy.config.source.api.Api(category, operation, parameters)

API parameter management.

snippy.config.source.base

Service

Configuration source base class.

class snippy.config.source.base.ConfigSourceBase(derived)

Base class for configuration sources.

init_conf(parameters)

Initialize configuration parameters.

Configuration can be read from command line interface or received from API query. It is also possible to configure for example server and storage parameters with environment variables. The precedence of configuration is:

  1. Command line option.

  2. Environment variable.

  3. Hard coded default.

Parameters

parameters (dict) – Parameters from configuration source.

property category

Get content category.

property data

Get content data.

property brief

Get content brief.

property description

Get content description.

property name

Get content name.

property groups

Get content groups.

property tags

Get content tags.

Get content links.

property source

Get content source.

property versions

Get content versions.

property languages

Get content languages.

property filename

Get content filename attribute.

property operation_files

Get operation filenames from the --file option.

property sall

Get ‘search all’ keywords.

property scat

Get content categories.

property stag

Get ‘search tag’ keywords.

property sgrp

Get ‘search groups’ keywords.

property search_filter

Get search regexp filter.

property search_limit

Get search result limit.

property search_offset

Get search offset from start.

property sort_fields

Get sorted fields.

property remove_fields

Get removed fields.

property reset_fields

Get reset fields.

property run_server

Get bool value that tells if Snippy server is run.

property server_base_path_rest

Get REST API base path.

property server_host

Get server host IP and port

property identity

Get content identity.

classmethod read_env(option, default)

Read parameter from optional environment variable.

Read parameter value from environment variable or return given default value. Environment variable names follow the same command line option naming convesion with modifications:

  1. Leading hyphens are removed.

  2. Option casing is converted to full upper case.

  3. Hyphens are replaced with underscores.

  4. SNIPPY_ prefix is added,

For example corresponding environment variable for the --server-host command line option is SNIPPY_SERVER_HOST.

Parameters

  • option (str) – Command line option.

  • default – Default value.

Returns

Same command line option name as received with value.

Return type

tuple

classmethod read_arg(option, default, args)

Read command line argument directly from sys.argv.

This is intenden to be used only in special cases that are related to debug options. The debug options are required for example to print logs before parsing command line arguments.

This function supports only bool and integer values because there are currently no other use cases.

This follows the standard command option parsing precedence:

  1. Command line option.

  2. Environment variable.

  3. Hard coded default.

Parameters

  • option (string) – Command line option.

  • default – Default value if option is not configured.

  • args (list) – Argument list received from command line.

Returns

Value for the command line option.

Return type

int,bool

get_plugin_short_names()

Get plugin short names.

The short names are used when user or client identifies a plugin.

The plugins are idenfied with prefix snippy- in the plugin name. The plugins are stored and operated without the snippy- prefix.

Returns

List of plugin names.

Return type

tuple

read_plugins(args)

Read all plugins.

Reading of plugins all the time is too slow. This is a problem with tests that slow down too much in case this would be read always.

Returns

Distribution objects from importlib_metadata.

Return type

dict

snippy.content.parser

Service

Parser class offers a parser to extract content fields from text source.

class snippy.content.parser.Parser(filetype, timestamp, source, collection)

Parse content attributes from text source.

read()

Read content attributes from text source.

Text source specific parser is run against the provided text string. The text source can be either the tool specific text or Markdown template.

snippy.content.parsers.base

Service

Content parser base class offers basic parsing methods.

class snippy.content.parsers.base.ContentParserBase

Base class for text content parser.

classmethod format_data(category, value)

Convert content data to utf-8 encoded tuple of lines.

Content data is stored as a tuple with one line per element.

All but solution data is trimmed from right for every line. In case of solution data, it is considered that user wants to leave it as is. Solutions are trimmed only so that there will be only one newline at the end of the solution data.

Any value including empty string is considered as a valid data.

Parameters

  • category (str) – Content category.

  • value (str,list) – Content data in string or list.

Returns

Tuple of utf-8 encoded unicode strings.

Return type

tuple

classmethod format_string(value)

Convert content string value to utf-8 encoded string.

Parameters

value (str,list,tuple) – Content field value in string, list or tuple.

Returns

Utf-8 encoded unicode string.

Return type

str

classmethod format_search_keywords(value)

Convert search keywords to utf-8 encoded tuple.

If the value is None it indicates that the search keywords were not given at all.

The keyword list may be empty or it can contain empty string. Both cases must be evaluated to ‘match any’.

Parameters

value (str,list,tuple) – Search keywords in string, list or tuple.

Returns

Tuple of utf-8 encoded keywords.

Return type

tuple

classmethod format_list(keywords, unique=True, sort_=True)

Convert list of keywords to utf-8 encoded list of strings.

Parse user provided keyword list. The keywords are for example groups, tags, search all keywords or versions.

It is possible to use string or list context for the given keywords. In case of list context, each element in the list is still split separately.

The keywords are split in word boundaries.

A dot is a special case. It is allowed for a regexp to match and search all records. The dot is also used in cases wuhere user wants to search strings with “docker.stop”. This matches to “docker stop” and it does not search separated words “docker” or “stop”.

Content versions field must support specific mathematical operators that do not split the keyword.

Parameters

  • keywords (str,list,tuple) – Keywords in string, list or tuple.

  • unique (bool) – Return unique keyword values.

  • sort_ (bool) – Return sorted keywords.

Returns

Tuple of utf-8 encoded keywords.

Return type

tuple

Convert links to utf-8 encoded list of links.

Parse user provided link list. Because URL and keyword have different forbidden characters, the methods to parse keywords are similar but still they are separated. URLs can be separated only with space, bar or newline. Space and bar characters are defined ‘unsafe characters’ in URL character set [1]. The newline is always URL encoded so it does not appear as newline.

The newline is supported here because that is used to separate links in text input.

Links are not sorted. The reason is that the sort is done based on content category. The content category is not know for sure when command options are parsed in this class. For this reason, the sort is always made later in the Resource when content category is known for sure.

[1] https://perishablepress.com/stop-using-unsafe-characters-in-urls/

Parameters

  • links (str,list,tuple) – Links in a string, list or tuple.

  • unique (bool) – Return unique keyword values.

Returns

Tuple of utf-8 encoded links.

Return type

tuple

classmethod format_filenames(filenames)

Convert filenames to utf-8 encoded list of filenames.

Parse user provided list of filenames. The filenames are separated with whitespaces. The filenames can contain any characters, even whitespaces. Because of this, the given filenames must be a list of filenames where each file can contain whitespaces.

Parameters

filenames (list,tuple) – Filenames in a list or tuple.

Returns

Tuple of utf-8 encoded filenames.

Return type

tuple

classmethod format_versions(versions)

Convert versions to utf-8 encoded list of version.

Only specific operators between key value versions are allowed.

Parameters

versions (str,list,tuple) – Versions in a string, list or tuple.

Returns

Tuple of utf-8 encoded versions.

Return type

tuple

classmethod parse_groups(category, regexp, text)

Parse content groups from text string.

There is always a default group added into the content group field.

Parameters

  • category (str) – Content category.

  • regexp (re) – Compiled regexp to search groups.

  • text (str) – Content text string.

Returns

Tuple of utf-8 encoded groups.

Return type

tuple

Parse content links from text string.

Parameters

  • category (str) – Content category.

  • regexp (re) – Compiled regexp to search links.

  • text (str) – Content text string.

Returns

Tuple of utf-8 encoded links.

Return type

tuple

classmethod parse_versions(category, regexp, text)

Parse content versions from text string.

Version strings are validated. Only versions which pass the validation rules are stored. The rules allow only specific operators between key value pairs.

Parameters

  • category (str) – Content category.

  • regexp (re) – Compiled regexp to search versions.

  • text (str) – Content text string.

Returns

Tuple of utf-8 encoded versions.

Return type

tuple

classmethod remove_template_fillers(content)

Remove tags and examples from content.

There are examples and tags in content templates that need to be removed before further processing the content. This method removes all the unnecessary tags and examples that are set to help user to fill a content template.

The received content can be text for Markdown based.

Parameters

content (str) – Content text or Markdown string.

Returns

String without content fillers.

Return type

str

classmethod to_unicode(value, strip_lines=True)

Convert value to utf-8 coded unicode string.

If the value is already an unicode character, it is assumed that it is a valid utf-8 encoded unicode character.

The conversion quarantees one newline at the end of string.

Parameters

  • value (str,list,tuple) – Value in a string, list or tuple.

  • strip_lines (bool) – Defines if all lines are stripped.

Returns

Utf-8 encoded unicode string.

Return type

str

read_brief(category, text)

Read content brief attribute.

Parameters

  • category (str) – Content category.

  • text (str) – Content string.

Returns

Utf-8 encoded unicode brief string.

Return type

str

read_description(category, text)

Read content description attribute.

Parameters

  • category (str) – Content category.

  • text (str) – Content string.

Returns

Utf-8 encoded unicode description string.

Return type

str

read_groups(category, text)

Read content groups attribute.

Parameters

  • category (str) – Content category.

  • text (str) – Content string.

Returns

Tuple of utf-8 encoded groups.

Return type

tuple

Read content links attribute.

Parameters

  • category (str) – Content category.

  • text (str) – Content string.

Returns

Tuple of utf-8 encoded links.

Return type

tuple

read_versions(category, text)

Read content versions attribute.

Parameters

  • category (str) – Content category.

  • text (str) – Content string.

Returns

Tuple of utf-8 encoded versions.

Return type

tuple

read_meta_value(category, key, text)

Read content metadata value from a text string.

Parameters

  • category (str) – Content category.

  • metadata (str) – Metadata to be read.

  • text (str) – Content text string.

Returns

Utf-8 encoded unicode string.

Return type

str

snippy.content.parsers.text

Service

Content parser for text content.

class snippy.content.parsers.text.ContentParserText(timestamp, text, collection)

Parse content from text template.

read_collection()

Read collection from the given text source.

snippy.content.parsers.mkdn

Service

Content parser for Markdown content.

class snippy.content.parsers.mkdn.ContentParserMkdn(timestamp, text, collection)

Parse content from Markdown template.

read_collection()

Read collection from the given Markdown source.

The content is split based on reserved syntax. The reserved syntax to separate two different contents is a string ‘—’ that must be at the beginning of a (multiline) string and have a newline immediately after the reserved string.

snippy.content.parsers.dict

Service

Content parser for YAML and JSON content.

class snippy.content.parsers.dict.ContentParserDict(timestamp, dictionary, collection)

Parse content from dictionary.

read_collection()

Read collection from the given dictionary source.

snippy.storage.storage

Service

Storage class offers database agnosting storage services. This abstracts the actual database solution from rest of the implementation.

class snippy.storage.storage.Storage

Storage management for content.

create(collection)

Create new content.

Parameters

collection (Collection) – Content container to be stored into database.

search(scat=, sall=, stag=, sgrp=, search_filter=None, uuid=None, digest=None, identity=None, data=None)

Search content.

Parameters

  • scat (tuple) – Search category keyword list.

  • sall (tuple) – Search all keyword list.

  • stag (tuple) – Search tag keyword list.

  • sgrp (tuple) – Search group keyword list.

  • search_filter (str) – Regexp filter to limit search results.

  • uuid (str) – Search specific uuid or part of it.

  • digest (str) – Search specific digest or part of it.

  • identity (str) – Search specific digest or UUID or part of them.

  • data (str) – Search specific content data or part of it.

Returns

Search result in Collection of content.

Return type

Collection

uniques(field, scat, sall, stag, sgrp)

Get unique values for given field.

Parameters

field (str) – Content field which unique values are read.

Returns

List of unique values for give field.

Return type

tuple

update(digest, resource)

Update resource specified by digest.

Parameters

  • digest (str) – Content digest that is udpated.

  • resource (Resource) – A single Resource() container that contains updates.

delete(digest)

Delete content.

Parameters

digest (str) – Content digest that is deleted.

export_content(scat=)

Export content.

Parameters

scat (tuple) – Search category keyword list.

import_content(collection)

Import content.

Parameters

collection (Collection) – Content container to be imported into database.

disconnect()

Disconnect storage.

debug()

Debug storage.

snippy.storage.database

Service

SqliteDb class offers database implementation for the Storage class.

snippy.storage.database

alias of snippy.storage.database