Development¶
Quick Start¶
For the development, you can clone the repository and run the setup for Python virtual environment like below:
git clone https://github.com/heilaaks/snippy.git
mkvirtualenv snippy
make dev
The basic commands to run and test are:
python3 runner create -c 'docker rm $(docker ps -a -q)' -b 'Remove all docker containers' -t docker,container,cleanup
make test
make lint
make coverage
make docs
make clean
Python Virtual Environment¶
You can install the Python virtual environment wrapper like below:
mkdir -p ${HOME}/devel/python-virtualenvs
sudo pip3 install virtualenvwrapper
virtualenv --version
export WORKON_HOME=${HOME}/devel/python-virtualenvs # Add to ~/.bashrc
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3 # Add to ~/.bashrc
source /usr/bin/virtualenvwrapper.sh # Add to ~/.bashrc
mkvirtualenv snippy
Example commands to operate the virtual environment are below. More information can be found from the Python virtualenvwrapper command reference documentation.
lssitepackages
lsvirtualenv
deactivate
workon snippy
rmvirtualenv snippy
Pylint¶
The Pylint rc file can be generated for the very first time like:
pylint --generate-rcfile > tests/pylint/pylint-snippy.rc
Releasing¶
Preparations
1. Update version number in meta.py. 1. Update the CHANGELOG.rst. 1. Updated README.rst and documentation.
Run tests
make test make lint make docs tox python setup.py check --restructuredtext
Test installation
make clean make clean-db pip uninstall snippy pip install . --user snippy --help snippy import --defaults snippy import --defaults --solutions snippy import --defaults --references snippy search --sall docker rm -f ${HOME}/devel/temp/snippy.db snippy import --defaults --storage-path ${HOME}/devel/temp snippy import --defaults --solutions --storage-path ${HOME}/devel/temp snippy import --defaults --references --storage-path ${HOME}/devel/temp snippy --server --storage-path ${HOME}/devel/temp --port 8080 --ip 127.0.0.1 & curl -s -X GET "http://127.0.0.1:8080/snippy/api/app/v1/snippets?sall=docker&limit=2" -H "accept: application/vnd.api+json" | python -m json.tool pkill snippy
Test with PyPI
# Install the tool into test PyPI. make clean make clean-db python setup.py sdist bdist_wheel upload -r testpypi # repository: https://test.pypi.org/legacy/ in ~/.pypirc sudo pip uninstall snippy -y pip uninstall snippy -y # Set path to local user bin and install the tool from test PyPI. PATH=$HOME/.local/bin:$PATH pip install --user --index-url https://test.pypi.org/simple/ snippy snippy --help snippy import --defaults snippy import --defaults --solutions snippy search --sall docker # Run all example commands from top down from: snippy --help snippy --help examples # Uninstall pip uninstall snippy -y
Test with Docker
su make docker docker rm $(docker ps --all -q -f status=exited) docker images -q --filter dangling=true | xargs docker rmi docker images docker rmi heilaaks/snippy:v0.7.0 docker rmi docker.io/heilaaks/snippy:latest docker rmi docker.io/heilaaks/snippy:v0.7.0 docker run heilaaks/snippy --help docker run heilaaks/snippy search --sall docker docker run -d --net="host" --name snippy heilaaks/snippy --server --port 8080 --ip 127.0.0.1 -vv curl -s -X GET "http://127.0.0.1:8080/snippy/api/app/v1/snippets?sall=docker&limit=2" -H "accept: application/vnd.api+json" docker logs 632e97aa83fe docker stop 632e97aa83fe docker rm $(docker ps --all -q -f status=exited) docker run -d --net="host" --name snippy heilaaks/snippy --server --log-json -vv & curl -s -X GET "http://127.0.0.1:8080/snippy/api/app/v1/snippets?sall=docker&limit=2" -H "accept: application/vnd.api+json" docker logs f72b3902dd4b
Test with PyPy
# Default test box install PyPy 2.7 from dnf. sudo dnf install pypy export PYTHONPATH=/usr/lib64/python2.7/site-packages/ wget https://bootstrap.pypa.io/get-pip.py sudo pypy get-pip.py sudo pypy -m pip install codecov sudo pypy -m pip install logging_tree sudo pypy -m pip install mock sudo pypy -m pip install pytest sudo pypy -m pip install pytest-cov sudo pypy -m pip install pytest-mock sudo pypy -m pip install falcon sudo pypy -m pip install gunicorn sudo pypy -m pip install jsonschema pypy runner --help pypy runner --server -vv pypy -m pytest -x ./tests/test_*.py --cov snipp unset PYTHONPATH
Verify data in CHANGELOG.rst
- Update the CHANGELOG.rst release date if needed.
Make tag
git tag -a v0.8.0 -m "Add new content category resources" git push -u origin v0.8.0
Releas in PyPI
python setup.py sdist bdist_wheel twine upload dist/*
Test PyPI release
sudo pip uninstall snippy -y pip install snippy --user snippy --help snippy import --defaults snippy import --defaults --solutions snippy search --sall docker
Release in Docker Hub
su docker login docker.io docker images sudo docker tag 5dc22d1d3380 docker.io/heilaaks/snippy:v0.7.0 sudo docker tag 5dc22d1d3380 docker.io/heilaaks/snippy:latest sudo docker push docker.io/heilaaks/snippy:v0.7.0 sudo docker push docker.io/heilaaks/snippy:latest
Test Docker release
su docker rm $(docker ps --all -q -f status=exited) docker images -q --filter dangling=true | xargs docker rmi docker images docker rmi heilaaks/snippy:v0.7.0 docker rmi heilaaks/snippy:latest docker rmi docker.io/heilaaks/snippy:latest docker rmi docker.io/heilaaks/snippy:v0.7.0 docker run snippy --help docker run snippy search --sall docker docker run -d --net="host" --name snippy heilaaks/snippy --server --port 8080 --ip 127.0.0.1 -vv curl -s -X GET "http://127.0.0.1:8080/snippy/api/app/v1/snippets?sall=docker&limit=2" -H "accept: application/vnd.api+json" | python -m json.tool docker run -d --net="host" --name snippy heilaaks/snippy --server --log-json -vv curl -s -X GET "http://127.0.0.1:8080/snippy/api/app/v1/snippets?sall=docker&limit=2" -H "accept: application/vnd.api+json" | python -m json.tool
Release news
- Make new release in Github.
Modules¶
snippy.logger¶
Service
Logger class offers logger for each caller based on the given module name. The configuration is controlled by global settings that are inherited by every logger.
The effective log level for all the loggers created under the ‘snippy’ logger namespace is inherited from the root logger which controls the log level. This relies on that the module level logger does not set the level and it remains as NOTSET. This causes module level logger to propagate the log record to parent where it eventually reaches the ‘snippy’ top level namespace that is just below the ‘root’ logger.
Behaviour
By default, there are no logs printed to the users. This applies also to error logs.
There are two levels of logging verbosity. All logs are printed in full length without filters with the –debug option. The -vv (very verbose) option prints limited length log messages in lower case letters.
There are two formats for logs: text (default) and JSON. JSON logs can be enabled with –log-json option. A JSON log has more information fields than text formatted log. When -vv option is used with JSON logs, it truncates log message in the same way as with text logs.
Timestamps are in local time with text formatted logs. In case of JSON logs, the timestamp is in GMT time zone and it follows strictly the ISO8601 format. Both timestamps are in millisecond granularity.
The log levels are are from Python logger but they follow severity level names from RFC 5424. There is a custom security level reserved only for security events.
All logs include operation ID that uniquely identifies all logs within specific operation. The operation ID must be refreshed by logger user after each operation is completed or the method must be wrapped with @Logger.timeit decorator which takes care of the OID refreshing.
All logs including Gunicorn server logs, are formatted to match format defined in this logger.
All logs are printed to stdout.
Security
There is a custom security level above critical level. This log level must be used only when there is a suspected security related event.
There is a hard maximum for log messages length for safety and security reasons. This tries to prevent extremely long log messages which may cause problems for the server.
Rules
- Only OK or NOK with cause text must be printed with default settings.
- There must be no logs printed to user.
- There must be no exceptions printed to user.
- Exceptions logs are printed as INFO and all other logs as DEBUG.
- Variables printed in logs must be separated with colon.
- All other than error logs must be printed as lower case string.
- The –debug option must print logs without filters in full-length.
- The -vv option must print logs in lower case and one log per line.
- All external libraries must follow the same log format.
- All logs must be printed to stdout.
Examples
# Variable printed at the end of log message is separated with colon. 2018-06-03 19:20:54.838 snippy[5756] [d] [b339bab5]: configured option server: true # Variable printed in the middle of log message is separated colons and # space from both sides. The purpose is to provide possibility to allow # log message post processing and to parse variables from log messages. 2018-06-03 19:20:54.838 snippy[5756] [d] [b339bab5]: server ip: 127.0.0.1 :and port: 8080
-
class
snippy.logger.
Logger
¶ Global logging services.
-
classmethod
get_logger
(name='snippy.logger')¶ Get logger.
A custom logger adapater is returned to support custom level with additional logging parameters.
Parameters: name (str) – Name of the module that requests a Logger.
-
classmethod
configure
(config)¶ Set and update logger configuration.
Parameters: config (dict) – Logger configuration dictionary. Examples
>>> Logger.configure({'debug': True, >>> 'log_json': True, >>> 'log_msg_max': Logger.DEFAULT_LOG_MSG_MAX, >>> 'quiet': False, >>> 'very_verbose': False})
-
static
reset
()¶ Reset log level to default.
-
static
remove
()¶ Delete all logger handlers.
-
classmethod
refresh_oid
()¶ Refresh operation ID (OID).
The OID is used to separate logs within one operation. The helps post-processing of the logs by allowing for example querying all the logs in failing operation.
-
classmethod
print_status
(status)¶ Print status information like exit cause or server running.
Parameters: status (str) – Status to be printed on stdout.
-
static
timeit
(method=None, refresh_oid=False)¶ Time method by measuring it latency.
The operation ID (OID) is refreshed at the end.
Parameters: - method (str) – Name of the method calling the timeit.
- refresh_oid (bool) – Define if operation ID is refreshed or not.
-
static
remove_ansi
(message)¶ Remove all ANSI escape codes from given string.
Parameters: message (str) – Log message which ANSI escape codes are removed.
-
static
debug
()¶ Debug Logger by printing logging hierarchy.
-
classmethod
snippy.cause¶
Service
Cause class offers storage services for normal and error causes. The causes are stored in a list where user can get all the failues that happened for example during the operation.
All causes are operated with predefind constants for HTTP causes and short descriptions of the event.
-
class
snippy.cause.
Cause
¶ Cause code services.
-
classmethod
reset
()¶ Reset cause to initial value.
-
classmethod
push
(status, message)¶ Append cause to list.
Parameters: - status (str) – One of the predefined HTTP status codes.
- message (str) – Description of the cause.
Examples
>>> Cause.push(Cause.HTTP_CREATED, 'content created')
-
classmethod
is_ok
()¶ Test if errors were detected.
-
classmethod
http_status
()¶ Return the HTTP status.
-
classmethod
json_message
()¶ Return errors in JSON data structure.
-
classmethod
get_message
()¶ Return cause message.
-
classmethod
print_message
()¶ Print cause message.
-
classmethod
print_failure
()¶ Print only failure message.
-
classmethod
debug
()¶ Debug Cause.
-
classmethod
snippy.storage.storage¶
Service
Storage class offers database agnosting storage services. This abstracts the actual database solution from rest of the implementation.
-
class
snippy.storage.storage.
Storage
¶ Storage management for content.
-
create
(collection)¶ Create new content.
Parameters: collection (Collection) – Content container to be stored into database.
-
search
(category, sall=None, stag=None, sgrp=None, digest=None, data=None)¶ Search content.
Parameters: - category (str) – Content category.
- sall (tuple) – Search all keyword list.
- stag (tuple) – Search tag keyword list.
- sgrp (tuple) – Search group keyword list.
- digest (str) – Search specific digest or part of it.
- data (str) – Search specific content data or part of it.
-
update
(digest, resource)¶ Update resource specified by digest.
Parameters: - digest (str) – Content digest that is udpated.
- resource (Resource) – A single Resource() container that contains updates.
-
delete
(digest)¶ Delete content.
Parameters: digest (str) – Content digest that is deleted.
-
export_content
(category)¶ Export content.
Parameters: category (str) – Content category.
-
import_content
(collection)¶ Import content.
Parameters: collection (Collection) – Content container to be imported into database.
-
disconnect
()¶ Disconnect storage.
-
debug
()¶ Debug storage.
-
snippy.storage.sqlitedb¶
Service
SqliteDb class offers database implementation for the Storage class.
-
class
snippy.storage.sqlitedb.
SqliteDb
¶ Sqlite database management.
-
init
()¶ Initialize database.
-
disconnect
()¶ Close database connection.
-
insert
(collection)¶ Insert collection into database.
Parameters: collection (Collection) – Content container to be stored into database. Returns: Collection of inserted content. Return type: Collection
-
select
(category, sall=(), stag=(), sgrp=(), digest=None, data=None)¶ Select content based on search criteria.
Parameters: - category (str) – Content category.
- sall (tuple) – Search all keyword list.
- stag (tuple) – Search tag keyword list.
- sgrp (tuple) – Search group keyword list.
- digest (str) – Search specific digest or part of it.
- data (str) – Search specific content data or part of it.
Returns: Collection of selected content.
Return type: Collection
-
select_all
(category)¶ Select all content from specific category.
Parameters: category (str) – Content category. Returns: Collection of all content in database. Return type: Collection
-
update
(digest, resource)¶ Update existing content.
Parameters: - digest (str) – Content digest that is udpated.
- resource (Resource) – Stored content in
Resource()
container.
Returns: Collection of updated content.
Return type: Collection
-
delete
(digest)¶ Delete content based on given digest.
Parameters: digest (str) – Content digest that is deleted.
-
debug
()¶ Debug Sqlitedb.
-