Diadosi: new architecture. Goodbye Python?

I have been experiencing some rather significant irritation and frustration with python, here lately. While the syntax and language features are unparalleled by other languages (except maybe, Nim Lang?), there are things about it that just irk me. GIL, threading isn't real, dependencies & required compiler toolchain, slow execution speed, etc.

Problem 1!

Why is my python thread blocking the entire process?

My biggest problem so far is actually not the Execution Speed, however it is, the Threading is not really Threading issue. While the documentation and examples might make it seem like you can achieve a real multi-threaded (Not multi-process) application, you simply can't. For example: try and write a little script that has a function or class which is started as a thread, to monitor inotify events in a unix-like OS. Simply put, that thread will take 100% of the allotted CPU time the OS gives your script.

To solve this problem, I had to write two separate scripts which communicate over beanstalkd. While this is ultimately a better and more scale-able solution, it is way overkill for what I was trying to accomplish. (I could have used sub processes, but the way python forks, was causing some really weird issues when the sub-process tried to `shell exec` an expect script.)

Problem 2..

Why do I need a compiler to install python packages?

Dependencies, Freaking Dependencies!!!! Must I explain this more? I have an entire post dedicated to this issue. All summed up, it is just a royal pain in the butt. Requiring specific compiler version on windows, and at times I have had four (4) different versions of MSVC++ installed on my machine. It is just stupid. Please read Python is Great but has a Huge Flaw!

Problem 3...

Why is python so dang slow?

Believe it or not, this is really isn't that big of an issue for me currently, but as my client base grows, and I sell more licenses to my online services, this will become a much bigger problem. Servers are cheap, but not free. Using a resource hungry language and environment directly impact profit margins. I will make less money per client while using python.


How do we fix these issues?

I wish I had a real answer to this question. I don't know what we can do to motivate the Python core team to focus on these issues. Python comes with such a wealth of libraries, to do just about anything you can imagine. For this reason alone it might be worth continuing to use Python even without these issues fixed. But, I am seriously investigating Nim Lang as an alternative to writing my more demanding applications in.

For example; I wrote a custom Lisp query language for Diadosi. It is actually developed in Python, to perform CRUD like operations, and some more complicated queries, which would be difficult to implement in a normal function call based API, here is a sample:

# QLisp Example
(select Post 
    (match ('status' 'active') ('author' (in 'Don Duvall' 'Tazd\'ij' 'Anonymous' 'Admin')))
    (include-fields 'title' 'body' 'meta.keywords' 'meta.description' 'meta.tags')
    (limit 5)
    (sub-select Author (as 'authorData') (on Post.author Author.name)))

Parsing this, while easy in Python, thanks to a couple libraries, is just really slow. As it is now all of the QLisp lexer/parser/analyzer processing is done in a microservice which allows me to have multiple machines dedicated to building the queries and offloading the internal commands to the appropriate servers, gathering the results, packaging them up and shipping them back out as the result of a QLisp API call.

I am planning on adding session and account context for stored procedures to reduce this, however it is complicated by the multiple QLisp Engine instances running. A lot of this complexity can be avoided by developing this piece in another language which is compiled and actually fast.