Skip navigation

Category Archives: Personal Projects

Coming off from my last post about updating pip inside a virtualenv, I have, finally, realized what I’ve been doing wrong all this time: I use this nifty script called virtualenv burrito to get a virtualenv + virtualenvwrapper set-up.

I’ve been writing an install script for a new project I am working on. Being at such an early phase, the script is scrappy and will only probably work for machines configured exactly as mine. Not that I make a lot of customizations, but still it relies on the existence of virtualenvwrapper too much.

The thing is I could never get my install script to run properly since pip is broken in my virtualenvs. So just yesterday, I finally decided to devote some time to digging into why precisely this is happening.

Long story short, after a few hours digging around the Vogon poetry of bash scripting, I realized that I can’t find separate virtualenv and virtualenvwrapper installs in my system…because I used virtualenv burrito to install them.

So I deleted the .venvburrito directory in my home and rerun virtualenv burrito. And…tadddaaaahhh!

chad@scheherazade:~$ mkvirtualenv fresh
New python executable in /home/chad/.virtualenvs/fresh/bin/python
Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating /home/chad/.virtualenvs/fresh/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/chad/.virtualenvs/fresh/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/chad/.virtualenvs/fresh/bin/preactivate
virtualenvwrapper.user_scripts creating /home/chad/.virtualenvs/fresh/bin/postactivate
virtualenvwrapper.user_scripts creating /home/chad/.virtualenvs/fresh/bin/get_env_details
(fresh) chad@scheherazade:~$ 
(fresh) chad@scheherazade:~$ pip -V
pip 10.0.1 from /home/chad/.venvburrito/lib/python2.7/site-packages/pip-10.0.1-py2.7.egg/pip (python 2.7

What this does not address, however, are my old virtualenvs which are stuck on an outdated pip. But I guess it should be trivial to just delete those virtualenvs and create a new one, this time with an updated pip.

There’s this thing in life, where you have assumptions you neglect to state because you thought (assume) them to be inconsequential, then they turn out to be of great importance. This is all the more true when you work with computers.

I love Python. I think it is a very productive language for prototyping, pedagogy, and experimentation with a mature ecosystem to support you as you take your shit code ideas to production. However, in all my years using it, I could never get to upgrade pip, its preferred package manager, inside a virtualenv without problems. And that frustrates me.

The story starts with one of the first things you tell Python noobs before they pwn their own system and blame it all on hallowed Python: use virtualenvs. I’m not sure if this applies to Windows users (or if the concept of virtualenvs even applies to Windows users) but for *nix users, this advice makes sense since Python is often used by your system for other, more-important-than-your-side-projects, OS-y things it does. This is why Python often comes pre-packaged with the system and why you should use virtualenvs: the last thing you want is to upgrade the datetime library which your NTP daemon is using only to find out that said daemon relies on a particular quirk of that datetime library version to function properly. And so virtualenvs.

I do not intend to discuss how virtualenvs work, but suffice it to say that it does magic so that the libraries your projects need are isolated. If that whole story about NTP daemons and datetime above just flew over your head, at least appreciate how nice it is to have your dependencies isolated like that; alexandria could not see PyGame, and PyGame Objects could not see Flask, and yet I develop them on the same machine. It tingles that part of my brain which makes me put my glass/cup down at different spots on the table every time after I take a sip so that I end up with liquid circles touching each other but not overlapping. An ordered world is just so zen.

(Although I should note that I’ve long given up making PyGame work consistently with virtualenvs across the machines I use. Nowadays, I just containerize the whole thing to find almost the same peace of mind.)

Anyway, I’m beginning to digress. To proceed with the story, there comes a time when issuing pip commands inside a virtualenv results to the following warning:

(origin3) chad@scheherazade:~$ pip freeze
pbr==1.10.0
six==1.10.0
stevedore==1.18.0
virtualenv==15.0.3
virtualenv-clone==0.2.6
virtualenvwrapper==4.7.2
You are using pip version 8.1.2, however version 10.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

And, well, who does not want free updates? Not to mention when the software involved is not just free but also open-source. And have I told you that this free and open-source software is part of the ecosystem of my favorite programming language? Up-update and away, I say!

(origin3) chad@scheherazade:~$ pip install --upgrade pip
Collecting pip
  Using cached https://files.pythonhosted.org/packages/0f/74/ecd13431bcc456ed390b44c8a6e917c1820365cbebcb6a8974d1cd045ab4/pip-10.0.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 8.1.2
    Not uninstalling pip at /home/chad/.venvburrito/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg, outside environment /home/chad/.virtualenvs/origin3
Successfully installed pip-8.1.2
You are using pip version 8.1.2, however version 10.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Err…okay, that did not really work. But not to worry! The beauty of using FOSS is that help is almost always available online. Especially on StackOverflow, a pillar of modern software engineering. Which leads us to the following words of wisdom:

pip is just a PyPI package like any other; you could use it to upgrade itself the same way you would upgrade any package:

~Cairnarvon on StackOverflow

But, but…Cairnarvon good sir, said instructions not working is what exactly set me on this quest! Then again, no one else is complaining that the accepted answer does not really work so I guess I’m the idiot here.

Thankfully, there are more words of wisdom in StackOverflow. If one option does not work, you have options. Fortunate then that this idiot can read. Since “pip is just a PyPI package”, it stands to reason that you can also upgrade it using its totally-not-sexy sibling of a package manager, easy_install.

Now, the other thing you tell Python noobs, maybe even before you tell them about virtualenvs, is to use pip over easy_install because reasons. So at this point, I am conflicted: one of the most ingrained teachings in my being is to use pip over easy_install but I have also been told to use updated software packages as much as possible. The latter urge wins out so I end up doing naughty things…

(origin3) chad@scheherazade:~$ easy_install pip
Searching for pip
Best match: pip 10.0.1
Adding pip 10.0.1 to easy-install.pth file
Installing pip3.6 script to /home/chad/.virtualenvs/origin3/bin
Installing pip3 script to /home/chad/.virtualenvs/origin3/bin
Installing pip script to /home/chad/.virtualenvs/origin3/bin

Using /home/chad/.virtualenvs/origin3/lib/python3.5/site-packages
Processing dependencies for pip
Finished processing dependencies for pip

…with good results:

(origin3) chad@scheherazade:pydagogical$ pip --version
pip 10.0.1 from /home/chad/.virtualenvs/origin3/lib/python3.5/site-packages/pip (python 3.5)

Or maybe not.

Bolstered by my enlightenment that you can upgrade pip using easy_install, I proceed to spin up new virtualenvs, just to savor the feeling of having an upgraded pip at last.

(spam) chad@scheherazade:~$ pip freeze
Traceback (most recent call last):
  File "/home/chad/.virtualenvs/spam/bin/pip", line 7, in 
    from pip._internal import main
ImportError: No module named 'pip._internal'

Thankfully, as a Professional Software Engineer with Years in the Industry, I can confidently read and debug the source code of the tools I am using. So I figured out that I should change line 7 of

1
2
3
4
5
6
7
8
9
10
11
#!/home/chad/.virtualenvs/spam/bin/python3
 
# -*- coding: utf-8 -*-
import re
import sys
 
from pip._internal import main
 
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

to

from pip import main

And everything should work fine and dandy again…

…except for the pip version:

(spam) chad@scheherazade:~$ pip --version
pip 8.1.2 from /home/chad/.venvburrito/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg (python 3.5)

I remember first encountering this all the way back at 2012, under, I think, 10.04 Lucid Lynx. But back then, I had a potato for a computer–the kind which weeps against Eclipse–and Ubuntu isn’t as polished as it is now. But, for crying out loud, it is 2018, I am on machines with glorious i7 processors and at least 16GB of RAM and this shit is still a thing.

At this point, Python has gone from a language whose ecosystem just works to a time waster to rival all the times I had to manually type public static void main(String[] args) just to test a damn Java behavior. I call it quits and I’m just happy I even have virtualenvs with a working pip installation.

A couple of years back give or take, I resolved getting serious with my dream of becoming a guitar god. So I wrote an app that would help me practice playing/jamming. When I finally resolved to write Museic in Java, it occurred to me that the nature of the app makes it a good mobile app project. However, a few factors like my machine specs then, kept me to a hacky desktop app.

Fast forward to today, I’m a better guitarist but no close to being Jason Mraz or Ed Sheeran. However, I’d like to think that the capability of my personal machine has increased in greater proportion than my guitar playing skill. So it seemed to me that, at last, it is time to tackle that good mobile app project.

And tackle it I did! Despite a couple more setbacks that I won’t go into here, it is with pride that I present the first mobile app I exclusively wrote myself. And whereas most of my projects just live at GitHub, often not in a polished state, and being “deployed” on my local machine at most, this time I have something that definitely screams “finished product”: a page in the app store.

Museician at Google Play Store

This is incidentally the first time I tried using the built-in page screenshot tool in Chrome. Can’t say I’m very impressed.

Born out of my own frustrations playing the guitar, making this felt awesome in more ways than one. Of course it lessens my frustration when learning a piece on the guitar. And then there’s that fleeting high when you create something that was not there before, get something to work when you previously had no idea how. It is not perfect, not very pretty, but I am very pleased with myself for making this. Working in the industry, it is not every day when you can personally help yourself with a program you wrote, something you do not because you’ll get in trouble otherwise, and when it happens Big Data can go hang. I managed to help myself and improve my quality of guitar practice without having to set-up TensorFlow. I feel like 90% of start-ups can’t even say that. Wink wink cringe.

I have a few more things I’d like to achieve with Museician, though I also feel that I’m more likely to just get this to a really-polished state, maybe add just one more feature (if it does not take too long), and then probably jump to the next thing I want to learn. The bad thing about working in Software Engineering today is that there is always a lot to learn. The good thing about working in Software Engineering today is that you need to be constantly learning.

…encountering new errors along the way, as well.

chad@scheherazade:pydagogical$ python3 -m ai.ants
Ant 0 is a touring machine.
Traceback (most recent call last):
  File "/usr/lib/python3.5/random.py", line 253, in choice
    i = self._randbelow(len(seq))
  File "/usr/lib/python3.5/random.py", line 230, in _randbelow
    r = getrandbits(k)          # 0 <= r < 2**k
ValueError: number of bits must be greater than zero
 
During handling of the above exception, another exception occurred:
 
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/chad/kode/pydagogical/ai/ants.py", line 85, in <module>
    solver.solve()
  File "/home/chad/kode/pydagogical/ai/ants.py", line 57, in solve
    self.ant_tour(i)
  File "/home/chad/kode/pydagogical/ai/ants.py", line 41, in ant_tour
    next_city = self.get_random_next(self.antroutes[antno], current)
  File "/home/chad/kode/pydagogical/ai/ants.py", line 52, in get_random_next
    next_city = random.choice(choices[:int(len(choices) / 4)])
  File "/usr/lib/python3.5/random.py", line 255, in choice
    raise IndexError('Cannot choose from an empty sequence')
IndexError: Cannot choose from an empty sequence

Exception during an exception? Wow.

While technically not part of my 2016 Resolution, I also got this:

66th percentile on HackerRank

You may be thinking, “66th percentile! Three stars! For an Asian, your standards aren’t much.”

Well, considering that I have a full-time job and that just around this time last year I was below 20th percentile, I’m calling this a win. 😉 😉