Development

Quick Start

For the development, you can clone the repository and run the setup for Python virtual environment like below:

git clone https://github.com/heilaaks/snippy.git
mkvirtualenv snippy
make dev

The basic commands to run and test are:

python3 runner create -c 'docker rm $(docker ps -a -q)' -b 'Remove all docker containers' -t docker,container,cleanup
make test
make lint
make coverage
make docs
make clean

Python Virtual Environment

You can install the Python virtual environment wrapper like below:

mkdir -p ${HOME}/devel/python-virtualenvs
sudo pip3 install virtualenvwrapper
virtualenv --version
export WORKON_HOME=${HOME}/devel/python-virtualenvs # Add to ~/.bashrc
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3    # Add to ~/.bashrc
source /usr/bin/virtualenvwrapper.sh                # Add to ~/.bashrc
mkvirtualenv snippy

Example commands to operate the virtual environment are below. More information can be found from the Python virtualenvwrapper command reference documentation.

lssitepackages
lsvirtualenv
deactivate
workon snippy
rmvirtualenv snippy

Pylint

The Pylint rc file can be generated for the very first time like:

pylint --generate-rcfile > tests/pylint/pylint-snippy.rc

Releasing

  1. Preparations

    1. Update version number in meta.py. 1. Update the CHANGELOG.rst. 1. Updated README.rst and documentation.

  2. Run tests

    make test
    make lint
    make docs
    tox
    python setup.py check --restructuredtext
    
  3. Test installation

    make clean
    make clean-db
    pip uninstall snippy
    pip install . --user
    snippy --help
    snippy import --defaults
    snippy import --defaults --solutions
    snippy import --defaults --references
    snippy search --sall docker
    rm -f ${HOME}/devel/temp/snippy.db
    snippy import --defaults --storage-path ${HOME}/devel/temp
    snippy import --defaults --solutions --storage-path ${HOME}/devel/temp
    snippy import --defaults --references --storage-path ${HOME}/devel/temp
    snippy --server --storage-path ${HOME}/devel/temp --port 8080 --ip 127.0.0.1 &
    curl -s -X GET "http://127.0.0.1:8080/snippy/api/app/v1/snippets?sall=docker&limit=2" -H "accept: application/vnd.api+json" | python -m json.tool
    pkill snippy
    
  4. Test with PyPI

    # Install the tool into test PyPI.
    make clean
    make clean-db
    python setup.py sdist bdist_wheel upload -r testpypi # repository: https://test.pypi.org/legacy/ in ~/.pypirc
    sudo pip uninstall snippy -y
    pip uninstall snippy -y
    
    # Set path to local user bin and install the tool from test PyPI.
    PATH=$HOME/.local/bin:$PATH
    pip install --user --index-url https://test.pypi.org/simple/ snippy
    snippy --help
    snippy import --defaults
    snippy import --defaults --solutions
    snippy search --sall docker
    
    # Run all example commands from top down from:
    snippy --help
    snippy --help examples
    
    # Uninstall
    pip uninstall snippy -y
    
  5. Test with Docker

    su
    make docker
    docker rm $(docker ps --all -q -f status=exited)
    docker images -q --filter dangling=true | xargs docker rmi
    docker images
    docker rmi heilaaks/snippy:v0.7.0
    docker rmi docker.io/heilaaks/snippy:latest
    docker rmi docker.io/heilaaks/snippy:v0.7.0
    docker run heilaaks/snippy --help
    docker run heilaaks/snippy search --sall docker
    docker run -d --net="host" --name snippy heilaaks/snippy --server --port 8080 --ip 127.0.0.1 -vv
    curl -s -X GET "http://127.0.0.1:8080/snippy/api/app/v1/snippets?sall=docker&limit=2" -H "accept: application/vnd.api+json"
    docker logs 632e97aa83fe
    docker stop 632e97aa83fe
    docker rm $(docker ps --all -q -f status=exited)
    docker run -d --net="host" --name snippy heilaaks/snippy --server --log-json -vv &
    curl -s -X GET "http://127.0.0.1:8080/snippy/api/app/v1/snippets?sall=docker&limit=2" -H "accept: application/vnd.api+json"
    docker logs f72b3902dd4b
    
  6. Test with PyPy

    # Default test box install PyPy 2.7 from dnf.
    sudo dnf install pypy
    export PYTHONPATH=/usr/lib64/python2.7/site-packages/
    wget https://bootstrap.pypa.io/get-pip.py
    sudo pypy get-pip.py
    sudo pypy -m pip install codecov
    sudo pypy -m pip install logging_tree
    sudo pypy -m pip install mock
    sudo pypy -m pip install pytest
    sudo pypy -m pip install pytest-cov
    sudo pypy -m pip install pytest-mock
    sudo pypy -m pip install falcon
    sudo pypy -m pip install gunicorn
    sudo pypy -m pip install jsonschema
    pypy runner --help
    pypy runner --server -vv
    pypy -m pytest -x ./tests/test_*.py --cov snipp
    unset PYTHONPATH
    
  7. Verify data in CHANGELOG.rst

    1. Update the CHANGELOG.rst release date if needed.
  8. Make tag

    git tag -a v0.8.0 -m "Add new content category resources"
    git push -u origin v0.8.0
    
  9. Releas in PyPI

    python setup.py sdist bdist_wheel
    twine upload dist/*
    
  10. Test PyPI release

    sudo pip uninstall snippy -y
    pip install snippy --user
    snippy --help
    snippy import --defaults
    snippy import --defaults --solutions
    snippy search --sall docker
    
  11. Release in Docker Hub

    su
    docker login docker.io
    docker images
    sudo docker tag 5dc22d1d3380 docker.io/heilaaks/snippy:v0.7.0
    sudo docker tag 5dc22d1d3380 docker.io/heilaaks/snippy:latest
    sudo docker push docker.io/heilaaks/snippy:v0.7.0
    sudo docker push docker.io/heilaaks/snippy:latest
    
  12. Test Docker release

    su
    docker rm $(docker ps --all -q -f status=exited)
    docker images -q --filter dangling=true | xargs docker rmi
    docker images
    docker rmi heilaaks/snippy:v0.7.0
    docker rmi heilaaks/snippy:latest
    docker rmi docker.io/heilaaks/snippy:latest
    docker rmi docker.io/heilaaks/snippy:v0.7.0
    docker run snippy --help
    docker run snippy search --sall docker
    docker run -d --net="host" --name snippy heilaaks/snippy --server --port 8080 --ip 127.0.0.1 -vv
    curl -s -X GET "http://127.0.0.1:8080/snippy/api/app/v1/snippets?sall=docker&limit=2" -H "accept: application/vnd.api+json" | python -m json.tool
    docker run -d --net="host" --name snippy heilaaks/snippy --server --log-json -vv
    curl -s -X GET "http://127.0.0.1:8080/snippy/api/app/v1/snippets?sall=docker&limit=2" -H "accept: application/vnd.api+json" | python -m json.tool
    
  13. Release news

    1. Make new release in Github.

Modules

snippy.logger

Service

Logger class offers logger for each caller based on the given module name. The configuration is controlled by global settings that are inherited by every logger.

The effective log level for all the loggers created under the ‘snippy’ logger namespace is inherited from the root logger which controls the log level. This relies on that the module level logger does not set the level and it remains as NOTSET. This causes module level logger to propagate the log record to parent where it eventually reaches the ‘snippy’ top level namespace that is just below the ‘root’ logger.

Behaviour

By default, there are no logs printed to the users. This applies also to error logs.

There are two levels of logging verbosity. All logs are printed in full length without filters with the –debug option. The -vv (very verbose) option prints limited length log messages in lower case letters.

There are two formats for logs: text (default) and JSON. JSON logs can be enabled with –log-json option. A JSON log has more information fields than text formatted log. When -vv option is used with JSON logs, it truncates log message in the same way as with text logs.

Timestamps are in local time with text formatted logs. In case of JSON logs, the timestamp is in GMT time zone and it follows strictly the ISO8601 format. Both timestamps are in millisecond granularity.

The log levels are are from Python logger but they follow severity level names from RFC 5424. There is a custom security level reserved only for security events.

All logs include operation ID that uniquely identifies all logs within specific operation. The operation ID must be refreshed by logger user after each operation is completed or the method must be wrapped with @Logger.timeit decorator which takes care of the OID refreshing.

All logs including Gunicorn server logs, are formatted to match format defined in this logger.

All logs are printed to stdout.

Security

There is a custom security level above critical level. This log level must be used only when there is a suspected security related event.

There is a hard maximum for log messages length for safety and security reasons. This tries to prevent extremely long log messages which may cause problems for the server.

Rules

  1. Only OK or NOK with cause text must be printed with default settings.
  2. There must be no logs printed to user.
  3. There must be no exceptions printed to user.
  4. Exceptions logs are printed as INFO and all other logs as DEBUG.
  5. Variables printed in logs must be separated with colon.
  6. All other than error logs must be printed as lower case string.
  7. The –debug option must print logs without filters in full-length.
  8. The -vv option must print logs in lower case and one log per line.
  9. All external libraries must follow the same log format.
  10. All logs must be printed to stdout.

Examples

# Variable printed at the end of log message is separated with colon.
2018-06-03 19:20:54.838 snippy[5756] [d] [b339bab5]: configured option server: true

# Variable printed in the middle of log message is separated colons and
# space from both sides. The purpose is to provide possibility to allow
# log message post processing and to parse variables from log messages.
2018-06-03 19:20:54.838 snippy[5756] [d] [b339bab5]: server ip: 127.0.0.1 :and port: 8080
class snippy.logger.Logger

Global logging services.

classmethod get_logger(name='snippy.logger')

Get logger.

A custom logger adapater is returned to support custom level with additional logging parameters.

Parameters:name (str) – Name of the module that requests a Logger.
classmethod configure(config)

Set and update logger configuration.

Parameters:config (dict) – Logger configuration dictionary.

Examples

>>> Logger.configure({'debug': True,
>>>                   'log_json': True,
>>>                   'log_msg_max': Logger.DEFAULT_LOG_MSG_MAX,
>>>                   'quiet': False,
>>>                   'very_verbose': False})
static reset()

Reset log level to default.

static remove()

Delete all logger handlers.

classmethod refresh_oid()

Refresh operation ID (OID).

The OID is used to separate logs within one operation. The helps post-processing of the logs by allowing for example querying all the logs in failing operation.

classmethod print_status(status)

Print status information like exit cause or server running.

Parameters:status (str) – Status to be printed on stdout.
static timeit(method=None, refresh_oid=False)

Time method by measuring it latency.

The operation ID (OID) is refreshed at the end.

Parameters:
  • method (str) – Name of the method calling the timeit.
  • refresh_oid (bool) – Define if operation ID is refreshed or not.
static remove_ansi(message)

Remove all ANSI escape codes from given string.

Parameters:message (str) – Log message which ANSI escape codes are removed.
static debug()

Debug Logger by printing logging hierarchy.

snippy.cause

Service

Cause class offers storage services for normal and error causes. The causes are stored in a list where user can get all the failues that happened for example during the operation.

All causes are operated with predefind constants for HTTP causes and short descriptions of the event.

class snippy.cause.Cause

Cause code services.

classmethod reset()

Reset cause to initial value.

classmethod push(status, message)

Append cause to list.

Parameters:
  • status (str) – One of the predefined HTTP status codes.
  • message (str) – Description of the cause.

Examples

>>> Cause.push(Cause.HTTP_CREATED, 'content created')
classmethod is_ok()

Test if errors were detected.

classmethod http_status()

Return the HTTP status.

classmethod json_message()

Return errors in JSON data structure.

classmethod get_message()

Return cause message.

classmethod print_message()

Print cause message.

classmethod print_failure()

Print only failure message.

classmethod debug()

Debug Cause.

snippy.storage.storage

Service

Storage class offers database agnosting storage services. This abstracts the actual database solution from rest of the implementation.

class snippy.storage.storage.Storage

Storage management for content.

create(collection)

Create new content.

Parameters:collection (Collection) – Content container to be stored into database.
search(category, sall=None, stag=None, sgrp=None, digest=None, data=None)

Search content.

Parameters:
  • category (str) – Content category.
  • sall (tuple) – Search all keyword list.
  • stag (tuple) – Search tag keyword list.
  • sgrp (tuple) – Search group keyword list.
  • digest (str) – Search specific digest or part of it.
  • data (str) – Search specific content data or part of it.
update(digest, resource)

Update resource specified by digest.

Parameters:
  • digest (str) – Content digest that is udpated.
  • resource (Resource) – A single Resource() container that contains updates.
delete(digest)

Delete content.

Parameters:digest (str) – Content digest that is deleted.
export_content(category)

Export content.

Parameters:category (str) – Content category.
import_content(collection)

Import content.

Parameters:collection (Collection) – Content container to be imported into database.
disconnect()

Disconnect storage.

debug()

Debug storage.

snippy.storage.sqlitedb

Service

SqliteDb class offers database implementation for the Storage class.

class snippy.storage.sqlitedb.SqliteDb

Sqlite database management.

init()

Initialize database.

disconnect()

Close database connection.

insert(collection)

Insert collection into database.

Parameters:collection (Collection) – Content container to be stored into database.
Returns:Collection of inserted content.
Return type:Collection
select(category, sall=(), stag=(), sgrp=(), digest=None, data=None)

Select content based on search criteria.

Parameters:
  • category (str) – Content category.
  • sall (tuple) – Search all keyword list.
  • stag (tuple) – Search tag keyword list.
  • sgrp (tuple) – Search group keyword list.
  • digest (str) – Search specific digest or part of it.
  • data (str) – Search specific content data or part of it.
Returns:

Collection of selected content.

Return type:

Collection

select_all(category)

Select all content from specific category.

Parameters:category (str) – Content category.
Returns:Collection of all content in database.
Return type:Collection
update(digest, resource)

Update existing content.

Parameters:
  • digest (str) – Content digest that is udpated.
  • resource (Resource) – Stored content in Resource() container.
Returns:

Collection of updated content.

Return type:

Collection

delete(digest)

Delete content based on given digest.

Parameters:digest (str) – Content digest that is deleted.
debug()

Debug Sqlitedb.