python

vscode python – No module named ‘src’

I've had some issues with running python code in vscode.

I have a project folder.
The code is in the "src" folder running a file one level below

the error

my terminal always said:

    from src.mymodule import *
ModuleNotFoundError: No module named 'src'

The solution

ctrl + p

Preferences: Open Settings (JSON)

append

"terminal.integrated.env.linux": {
    "PYTHONPATH": "${workspaceFolder}/src:${env:PYTHONPATH}"
  },
  "terminal.integrated.env.osx": {
    "PYTHONPATH": "${workspaceFolder}/src:${env:PYTHONPATH}"
  },
  "terminal.integrated.env.windows": {
    "PYTHONPATH": "${workspaceFolder}/src;${env:PYTHONPATH}"
  },

One last step:

close the terminal running it
press on run code again

The issue is discussed here:

https://github.com/microsoft/vscode-python/issues/11875

Write Clean Javascript Code

A awesome resouce on writing clean javascript code
https://github.com/ryanmcdermott/clean-code-javascript

based on Robert C. Martin's book
Clean Code: A Handbook of Agile Software Craftsmanship

"The only valid measurement of code quality is WTFs/minute"

Variables

  • Use meaningful and pronounceable variable names
  • Use the same vocabulary for the same type of variable
  • Use searchable names
  • Use explanatory variables
  • Avoid Mental Mapping
  • Don't add unneeded context
  • Use default arguments instead of short circuiting or conditionals

Functions

  • Function arguments (2 or fewer ideally)
  • Functions should do one thing
  • Function names should say what they do
  • Functions should only be one level of abstraction
  • Remove duplicate code
  • Set default objects with Object.assign
  • Don't use flags as function parameters
  • Avoid Side Effects
  • Don't write to global functions
  • Favor functional programming over imperative programming
  • Encapsulate conditionals
  • Avoid negative conditionals
  • Avoid conditionals
  • Avoid type-checking
  • Don't over-optimize
  • Remove dead code

Objects and Data Structures

  • Use getters and setters
  • Make objects have private members

Classes

  • Prefer ES2015/ES6 classes over ES5 plain functions
  • Use method chaining
  • Prefer composition over inheritance

SOLID

  • Single Responsibility Principle (SRP)
  • Open/Closed Principle (OCP)
  • Liskov Substitution Principle (LSP)
  • Interface Segregation Principle (ISP)
  • Dependency Inversion Principle (DIP)

Testing

  • Single concept per test

Concurrency

  • Use Promises, not callbacks
  • Async/Await are even cleaner than Promises

Error Handling

  • Don't ignore caught errors
  • Don't ignore rejected promises

Formatting

  • Use consistent capitalization
  • Function callers and callees should be close

Comments

  • Only comment things that have business logic complexity.
  • Don't leave commented out code in your codebase
  • Don't have journal comments
  • Avoid positional markers

Microservices, APIs, SOA

I found a great article on "Microservices, SOA, and APIs: Friends or enemies?"
check it out.. really easy to understand the differences

https://developer.ibm.com/tutorials/1601_clark-trs/

A little bit about Microservices:

It's often said.. complex code is spaghetti code (with meatballs = OOP), well structured layered code is lasagne code.. and microservices ravioli code.

  • Complex Spaghetti Code
  • Well Layerd Code
  • Ravioli Code ( Microservices )

On bigger applications.. a well thought out monolithic modular app might not be enough or at least not performant enough.

Big companies like Netflix talk about it.
https://www.nginx.com/blog/microservices-at-netflix-architectural-best-practices/

The millions of users require:

  • fast startup times,
  • easy and reactive ui,
  • working movies (main functionality),
  • no downtimes and
  • always new features/updates.

The solution to this of course is a microservice application

Thoughts before creating microservice instead of monolithic applications:

  • Do you need to scale? How much concurrent users do you have?
  • Does your application or several parts really need to have a high-availability?
  • Do you develop in separate teams or is everyone on one team?

Benefits

  • Scalability:
  • Microservices can be monitored really well, giving you metrics to find services with high hardware requirements or faulting services
  • run multiple instances for particular services, making reaction times really fast by also giving you a high availability (in case they crash)
  • You can run/scale on multiple Servers / PaaS Instances (Pivotal, AWS, ...) with different spec requirements depending on what your microservices needs
  • Improved Isolation:
  • every Microservice should be one logical part and thus can be updated or refactored or exchanged separate without affecting the others
  • By doing only one particular job (bounded context).. your microservices become easier to read, test and optimize
  • You can use multiple programming languages in one project. eg. node.js, ruby, c#, python, c, R.. in the end you talk via API
  • Background Services:
  • You can run background jobs, which wait for events to start working. they can do things without your interaction.

Disadvantages

  • Distributed System Complexity (Logging, Monitoring, Network, Msg Queues, Multiple Instances, )
  • with more Services, more parts can fail
  • Architecture causes network-load and can be slowed down by network-latency
  • May have double code for same helper functionalities

Programming Services it is highly recommended to apply the principles of

12 Factor Apps

https://12factor.net/

  • Code is Version controlled with many deploys [Codebase]
  • Dependencies are declared and isolated in a manifest [Dependencies]
  • Configurations are stored in ENVironment variables, not config files [Config]
  • Service are attached Resources, which can be consumed over Network (local or 3rd Party) and can be detached at will [Backing Service]
  • Build and Run stages are seperated. Providing a faster, tested and better Release Management. [Build, release, run]
  • Services are Stateless. They share nothing. They only use persistent data from backing stores. Thus they can easily be scaled. [Process]
  • To fit the concurrency needs, services are exported via Port Binding. [Concurrency]
  • Processes should be designed to be disposable without hassle [Disposable]
  • You should have a similar Developement and Production environment, designed for continuous integration. [Dev/Prod parity]
  • Logs should be treated as Event Streams. The App should never be concerned of storing logs. [Logs]
  • Management Tasks running as Processes: no local terminal, no direct db access. [Admin processes]

Terminator won’t start

I ran into a stupid error.

My terminal client Terminator wouldn't start on ubuntu and I didn't get any no error either.

I started it via terminal. Here I got the following error:

/usr/bin/terminator
 except (KeyError,ValueError), ex

I found the problem and solution
python3 was changed to my default system python

=> changed it back to 2.7, everything works again

I use update-alternatives

you can set up things easily. eg. python

check if it's maybe already set up for python

# CHECK if setup for python
update-alternatives --list python

otherwise set up the alternatives

# SETUP ALTERNATIVE 1
update-alternatives --install /usr/bin/python python /usr/bin/python2.7 1

# SETUP ALTERNATIVE 2
update-alternatives --install /usr/bin/python python /usr/bin/python3.6 2

and configure the version you want to be used by default

# CONFIGURE
update-alternatives --config python

There are 2 choices for the alternative python (providing /usr/bin/python).
   Selection    Path                Priority   Status
 0            /usr/bin/python3.6   2         auto mode
 1            /usr/bin/python2.7   1         manual mode
 2            /usr/bin/python3.6   2         manual mode 
 Press  to keep the current choice[*], or type selection number:

useful python ressource: python101

Everything is on the web. I would even say there is quantity over quality.
That's why finding good ressources saves often lot's of time.

Here is a good site about python3:

http://python101.pythonlibrary.org/

It's a ebook to support the writer but you can also read online.
It encounters a lot of python things you should know.

Here's the TOC:

Part I - Learning the Basics
Chapter 1 - IDLE Programming
Chapter 2 - All About Strings
Chapter 3 - Lists, Tuples and Dictionaries
Chapter 4 - Conditional Statements
Chapter 5 - Loops
Chapter 6 - Python Comprehensions
Chapter 7 - Exception Handling
Chapter 8 - Working with Files
Chapter 9 - Importing
Chapter 10 - Functions
Chapter 11 - Classes

Part II - Learning from the Library
Chapter 12 - Introspection
Chapter 13 - The csv Module
Chapter 14 - configparser
Chapter 15 - Logging
Chapter 16 - The os Module
Chapter 17 - The email / smtplib Module
Chapter 18 - The sqlite Module
Chapter 19 - The subprocess Module
Chapter 20 - The sys Module
Chapter 21 - The threading module
Chapter 22 - Working with Dates and Time
Chapter 23 - The xml module

Part III - Intermediate Odds and Ends
Chapter 24 - The Python Debugger
Chapter 25 - Decorators
Chapter 26 - The lambda
Chapter 27 - Code Profiling
Chapter 28 - An Intro to Testing

Part IV - Tips, Tricks and Tutorials
Chapter 29 - Installing Packages
Chapter 30 - ConfigObj
Chapter 31 - Parsing XML with lxml
Chapter 32 - Python Code Analysis
Chapter 33 - The requests package
Chapter 34 - SQLAlchemy
Chapter 35 - virtualenv

Part V - Packaging and Distribution
Chapter 36 - Creating Modules and Packages
Chapter 37 - How to Add Your Code to PyPI
Chapter 38 - The Python egg
Chapter 39 - Python wheels
Chapter 40 - py2exe
Chapter 41 - bbfreeze
Chapter 42 - cx_Freeze
Chapter 43 - PyInstaller
Chapter 44 - Creating an Installer

Jupyter Notebooks for “Python Data Science Handbook”

I've found a GitHub Repo which offers the content from O'Reilly's Python Data Science Handbook - Essential Tools for Working with data (Released Nov 2016).

The following content in form of Jupyter Notebooks is to be found:

  1. IPython: Beyond Normal Python
  2. Introduction to NumPy
  3. Data Manipulation with Pandas
  4. Visualization with Matplotlib
  5. Machine Learning 
    Appendix: Figure Code

If you want to learn more about Data Science, this is certainly a good way.

https://github.com/dunovank/PythonDataScienceHandbook

Multi-threaded 7-Zip with Zstandard, Brotli, Lz4, Lz5 and Lizard Compression support

Great compression performance now available for use in windows and more
https://github.com/mcmilk/7-Zip-zstd

You can also get there the multithreading c library for Brotli, Lizard, LZ4, LZ5 and ZStandard used in it
https://github.com/mcmilk/zstdmt

Python LZ4 lib

I've also found a good python multi-threaded lib for LZ4
https://github.com/Iotic-Labs/py-lz4framed

Easy to use

import lz4framed

compressed = lz4framed.compress(b'my normal text')

uncompressed = lz4framed.decompress(compressed)

file compression

with open('somefile.csv', 'wb') as f:
    # Context automatically finalises frame on completion, unless an exception occurs
    with Compressor(f) as c:
        try:
            while (...):
               c.update(moreData)
        except Lz4FramedNoDataError:
            pass

file decompression

with open('somefile.csv.lz4', 'rb') as f:
    try:
        for chunk in Decompressor(f):
           decoded.append(chunk)
    except Lz4FramedNoDataError:
        # Compress frame data incomplete - error case

command line utility

python3 -mlz4framed
USAGE: lz4framed (compress|decompress) (INFILE|-) [OUTFILE]

(De)compresses an lz4 frame. Input is read from INFILE unless set to '-', in
which case stdin is used. If OUTFILE is not specified, output goes to stdout.

Some benchmarks

Quick Benchmark: Gzip vs Bzip2 vs LZMA vs XZ vs LZ4 vs LZO

Tested were: Compression ratio, Compression time, Decompression time, Memory requirements on compression, Memory requirements on decompression, Time to compress, Time to decompress, Memory used, Compression ratio

https://catchchallenger.first-world.info/wiki/QuickBenchmark:GzipvsBzip2vsLZMAvsXZvsLZ4vsLZO