Skip navigation

Category Archives: Nerd Notes

Coming off from my last post about updating pip inside a virtualenv, I have, finally, realized what I’ve been doing wrong all this time: I use this nifty script called virtualenv burrito to get a virtualenv + virtualenvwrapper set-up.

I’ve been writing an install script for a new project I am working on. Being at such an early phase, the script is scrappy and will only probably work for machines configured exactly as mine. Not that I make a lot of customizations, but still it relies on the existence of virtualenvwrapper too much.

The thing is I could never get my install script to run properly since pip is broken in my virtualenvs. So just yesterday, I finally decided to devote some time to digging into why precisely this is happening.

Long story short, after a few hours digging around the Vogon poetry of bash scripting, I realized that I can’t find separate virtualenv and virtualenvwrapper installs in my system…because I used virtualenv burrito to install them.

So I deleted the .venvburrito directory in my home and rerun virtualenv burrito. And…tadddaaaahhh!

chad@scheherazade:~$ mkvirtualenv fresh
New python executable in /home/chad/.virtualenvs/fresh/bin/python
Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating /home/chad/.virtualenvs/fresh/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/chad/.virtualenvs/fresh/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/chad/.virtualenvs/fresh/bin/preactivate
virtualenvwrapper.user_scripts creating /home/chad/.virtualenvs/fresh/bin/postactivate
virtualenvwrapper.user_scripts creating /home/chad/.virtualenvs/fresh/bin/get_env_details
(fresh) chad@scheherazade:~$ 
(fresh) chad@scheherazade:~$ pip -V
pip 10.0.1 from /home/chad/.venvburrito/lib/python2.7/site-packages/pip-10.0.1-py2.7.egg/pip (python 2.7

What this does not address, however, are my old virtualenvs which are stuck on an outdated pip. But I guess it should be trivial to just delete those virtualenvs and create a new one, this time with an updated pip.

There’s this thing in life, where you have assumptions you neglect to state because you thought (assume) them to be inconsequential, then they turn out to be of great importance. This is all the more true when you work with computers.

I love Python. I think it is a very productive language for prototyping, pedagogy, and experimentation with a mature ecosystem to support you as you take your shit code ideas to production. However, in all my years using it, I could never get to upgrade pip, its preferred package manager, inside a virtualenv without problems. And that frustrates me.

The story starts with one of the first things you tell Python noobs before they pwn their own system and blame it all on hallowed Python: use virtualenvs. I’m not sure if this applies to Windows users (or if the concept of virtualenvs even applies to Windows users) but for *nix users, this advice makes sense since Python is often used by your system for other, more-important-than-your-side-projects, OS-y things it does. This is why Python often comes pre-packaged with the system and why you should use virtualenvs: the last thing you want is to upgrade the datetime library which your NTP daemon is using only to find out that said daemon relies on a particular quirk of that datetime library version to function properly. And so virtualenvs.

I do not intend to discuss how virtualenvs work, but suffice it to say that it does magic so that the libraries your projects need are isolated. If that whole story about NTP daemons and datetime above just flew over your head, at least appreciate how nice it is to have your dependencies isolated like that; alexandria could not see PyGame, and PyGame Objects could not see Flask, and yet I develop them on the same machine. It tingles that part of my brain which makes me put my glass/cup down at different spots on the table every time after I take a sip so that I end up with liquid circles touching each other but not overlapping. An ordered world is just so zen.

(Although I should note that I’ve long given up making PyGame work consistently with virtualenvs across the machines I use. Nowadays, I just containerize the whole thing to find almost the same peace of mind.)

Anyway, I’m beginning to digress. To proceed with the story, there comes a time when issuing pip commands inside a virtualenv results to the following warning:

(origin3) chad@scheherazade:~$ pip freeze
pbr==1.10.0
six==1.10.0
stevedore==1.18.0
virtualenv==15.0.3
virtualenv-clone==0.2.6
virtualenvwrapper==4.7.2
You are using pip version 8.1.2, however version 10.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

And, well, who does not want free updates? Not to mention when the software involved is not just free but also open-source. And have I told you that this free and open-source software is part of the ecosystem of my favorite programming language? Up-update and away, I say!

(origin3) chad@scheherazade:~$ pip install --upgrade pip
Collecting pip
  Using cached https://files.pythonhosted.org/packages/0f/74/ecd13431bcc456ed390b44c8a6e917c1820365cbebcb6a8974d1cd045ab4/pip-10.0.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 8.1.2
    Not uninstalling pip at /home/chad/.venvburrito/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg, outside environment /home/chad/.virtualenvs/origin3
Successfully installed pip-8.1.2
You are using pip version 8.1.2, however version 10.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Err…okay, that did not really work. But not to worry! The beauty of using FOSS is that help is almost always available online. Especially on StackOverflow, a pillar of modern software engineering. Which leads us to the following words of wisdom:

pip is just a PyPI package like any other; you could use it to upgrade itself the same way you would upgrade any package:

~Cairnarvon on StackOverflow

But, but…Cairnarvon good sir, said instructions not working is what exactly set me on this quest! Then again, no one else is complaining that the accepted answer does not really work so I guess I’m the idiot here.

Thankfully, there are more words of wisdom in StackOverflow. If one option does not work, you have options. Fortunate then that this idiot can read. Since “pip is just a PyPI package”, it stands to reason that you can also upgrade it using its totally-not-sexy sibling of a package manager, easy_install.

Now, the other thing you tell Python noobs, maybe even before you tell them about virtualenvs, is to use pip over easy_install because reasons. So at this point, I am conflicted: one of the most ingrained teachings in my being is to use pip over easy_install but I have also been told to use updated software packages as much as possible. The latter urge wins out so I end up doing naughty things…

(origin3) chad@scheherazade:~$ easy_install pip
Searching for pip
Best match: pip 10.0.1
Adding pip 10.0.1 to easy-install.pth file
Installing pip3.6 script to /home/chad/.virtualenvs/origin3/bin
Installing pip3 script to /home/chad/.virtualenvs/origin3/bin
Installing pip script to /home/chad/.virtualenvs/origin3/bin

Using /home/chad/.virtualenvs/origin3/lib/python3.5/site-packages
Processing dependencies for pip
Finished processing dependencies for pip

…with good results:

(origin3) chad@scheherazade:pydagogical$ pip --version
pip 10.0.1 from /home/chad/.virtualenvs/origin3/lib/python3.5/site-packages/pip (python 3.5)

Or maybe not.

Bolstered by my enlightenment that you can upgrade pip using easy_install, I proceed to spin up new virtualenvs, just to savor the feeling of having an upgraded pip at last.

(spam) chad@scheherazade:~$ pip freeze
Traceback (most recent call last):
  File "/home/chad/.virtualenvs/spam/bin/pip", line 7, in 
    from pip._internal import main
ImportError: No module named 'pip._internal'

Thankfully, as a Professional Software Engineer with Years in the Industry, I can confidently read and debug the source code of the tools I am using. So I figured out that I should change line 7 of

1
2
3
4
5
6
7
8
9
10
11
#!/home/chad/.virtualenvs/spam/bin/python3
 
# -*- coding: utf-8 -*-
import re
import sys
 
from pip._internal import main
 
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

to

from pip import main

And everything should work fine and dandy again…

…except for the pip version:

(spam) chad@scheherazade:~$ pip --version
pip 8.1.2 from /home/chad/.venvburrito/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg (python 3.5)

I remember first encountering this all the way back at 2012, under, I think, 10.04 Lucid Lynx. But back then, I had a potato for a computer–the kind which weeps against Eclipse–and Ubuntu isn’t as polished as it is now. But, for crying out loud, it is 2018, I am on machines with glorious i7 processors and at least 16GB of RAM and this shit is still a thing.

At this point, Python has gone from a language whose ecosystem just works to a time waster to rival all the times I had to manually type public static void main(String[] args) just to test a damn Java behavior. I call it quits and I’m just happy I even have virtualenvs with a working pip installation.

…encountering new errors along the way, as well.

chad@scheherazade:pydagogical$ python3 -m ai.ants
Ant 0 is a touring machine.
Traceback (most recent call last):
  File "/usr/lib/python3.5/random.py", line 253, in choice
    i = self._randbelow(len(seq))
  File "/usr/lib/python3.5/random.py", line 230, in _randbelow
    r = getrandbits(k)          # 0 <= r < 2**k
ValueError: number of bits must be greater than zero
 
During handling of the above exception, another exception occurred:
 
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/chad/kode/pydagogical/ai/ants.py", line 85, in <module>
    solver.solve()
  File "/home/chad/kode/pydagogical/ai/ants.py", line 57, in solve
    self.ant_tour(i)
  File "/home/chad/kode/pydagogical/ai/ants.py", line 41, in ant_tour
    next_city = self.get_random_next(self.antroutes[antno], current)
  File "/home/chad/kode/pydagogical/ai/ants.py", line 52, in get_random_next
    next_city = random.choice(choices[:int(len(choices) / 4)])
  File "/usr/lib/python3.5/random.py", line 255, in choice
    raise IndexError('Cannot choose from an empty sequence')
IndexError: Cannot choose from an empty sequence

Exception during an exception? Wow.

Ever since I took it to heart to write unit tests for my code, I’ve wondered how to unit test frontend functionality. Some will say (and this I remember from my early readings on unit testing) that frontend is not meant to be unit tested but, instead, should be end-to-end tested but I respectfully disagree.

(I’d admit though, that I’ve never groked the difference between unit tests and end-to-end tests. They’re all tests to me in that they ensure my implementation is up to spec and that my changes did not break existing functionality. The last part has saved my ass so much for Chess Templar.)

I disagree because frontend code has features that lends itself well to being a unit-testable component. Among them:

  • Creating DOM objects is creating objects. You should be able to assert the properties of the created objects given parameters x and y.
  • Frontend validation. You should be able to assert how your validators behave. All the more true as your validation logic becomes more complicated.
  • It is not uncommon to have a util.js file. These are utilities, possibly used across multiple files and by multiple developers. It better be tested!

Disappointingly, Javascript has remained trapped inside browsers. Sure enough, Node and company are working to do justice on that but the fact remains that not everyone is using Node and company and so their Javascript remain trapped inside browsers.

And it seems that learning how to unit test in Javascript is largely dependent on how you use Javascript. Are you using plain old jQuery? AngularJS, maybe? For my intents and purposes, I’m on plain old jQuery.

It shouldn’t come as a surprise then that I ended up choosing QUnit as my unit testing library. The docs are easy-to-understand enough. With that, you’ll be testing your Javascript in no time!

But then comes the part of automation. The project I am using this for is, quite frankly, small in scope and it would not cost me much neurons to keep in mind to fire up the HTML page of a test-runner I got from reading QUnit’s docs every time I modify my scripts. But a good programmer is a lazy programmer. So I tried to automate that the tests run at Travis as well1.

The easiest guide I can find comes from this StackOverflow answer. The answer assumes an Ant project and the original question is even for Atlassian Bamboo. However, with a few tweaks, I got it to work with the free set-up the internets afford me.

First, you’d have to install xvfb in your environment. Xvfb is crucial for making your tests headless—I’ve actually first encountered it when trying to automate Selenium tests.

What’s fantastic with CIs (done properly) is that your code gets a fresh instance to run on everytime. Though that also means setting up (meticulously) everytime as well; this means ensuring that xvfb and other dependencies like Firefox or a MySQL instance is running before your tests are ran. To do that in Travis, I had the following in my .travis.yml:

language: python
python:
    - "2.7"
before_script: "pip install -r test-requirements.txt"
install: "pip install -r requirements.txt"
script: "nosetests --with-xcoverage && ./run_js_tests"
before_install:
    - sudo apt-get update -qq
    - sudo apt-get install -y xvfb
    - sudo apt-get install -y firefox
    - sudo apt-get install -y mysql-server
    - sudo /etc/init.d/mysql start
    - sudo apt-get install -y mysql-client
    - mysql -uroot -e "CREATE DATABASE alexandria_test;"

Notes:

  • I had to install with the the -y flag. This automatically answers “yes” to all installation-related queries.
  • I had to start mysql manually.
  • I also need to create the test database manually. A few years ago I would be very uncomfortable with the hard-coded test DB name; I would want it to get the name from my config.

Next, we’re using JS Test Driver. There are a couple of Javascript libraries JAR file in the Google Code repo I linked that you need. If you are using git (as I am) and would prefer to submodule, I had the repo ported to git here. I have not yet found the time to submodule since my goal was just to get the set-up working properly.

Having all the required libraries, all the rest are just some configuration. The main call would then be,

java -jar ../lib/JsTestDriver.jar --config jsTestDriver.conf --port 4224 --browser $FIREFOX --tests all --testOutput $OUTPUT_DIR

Where --port is as configured in jsTestDriver.conf and $FIREFOX is the path to the local Firefox installation.

Next come the actual unit tests. If you have followed thus far, you should be able to see some logs related to Javascript tests in your CI server. However, if you followed QUnit’s documentation as it stands (and be able to run your tests from an HTML page test runner), you might notice that your CI reports of 0 tests found and 0 tests ran.

This is because the QUnitAdapter library in JS Test Driver features a different organization than the actual QUnit library. I had to modify my unit tests as in this commit. But in summary:

  • Don’t refer from QUnit. QUnitAdapter provides window-level (read: global) functions that are analogous to QUnit’s. So QUnit.test becomes test and QUnit.module just becomes module.
  • Similarly, forego the asserts. A notable exception is the equality assertion. QUnit’s assert has it as equal while QUnitAdapter has it as equals.
  • beforeEach becomes setup. Self-explanatory.

I figured this out because I read the source code of QUnitTestAdapter. There might be more quirks I haven’t found out yet but these should be enough to get you to testing functionality; the unit test code might not be as idiomatic as desired but hey, it works.

I’m still far from my ideal unit testing set-up for Javascript. All I’ve managed to achieve in this post is to get the tests to run in a CI environment. Notably, test failures for Javascript do not affect the result of the build. I’d also want to get the Javascript included in the coverage reports. And, maybe, reconcile QUnitTestAdapter with the actual constructs of plain QUnit.

  1. Interesting to note that jQuery does not have CI badges on its repo and that they use Node to automate. []

From my last post, I’m taking a little break from Chess as I turn my attention to a guitar I bought as a Christmas gift to myself. You see, I’m in the process of becoming a Guitar God, ala Jason Mraz, but that story is better reserved for my main blog no?

And well, since I can’t personally jam with Jason Mraz (or Chris Martin), I settle for MP3’s of his songs. The problem is, MP3s play abruptly without warning. There is no sufficient time between my hand pressing play and the song playing!

Jason Mraz Fanboy

And yes, I do have the CDs!

None of the MP3 players I know of is aspiring-musician-friendly as such. So, obviously, this makes it a nice candidate for a quick hack.

My requirements are simple:

  • Have a UI
  • Be able to play an MP3 file after a set countdown. And it has to be in MP3 format, hard requirement. If it is to be of any use with minimal hassles, it has to be MP3.

 

The first requirement makes it a good candidate for a web/JavaScript app. But then, as far as I know, JavaScript is not allowed access to the local filesystem for security purposes. So JavaScript (unfortunately) rules out the more-important requirement of the two I have.

The next obvious choice is Java since I’m quite used to utilizing Swing to develop desktop GUIs; in fact, I created the user interface of our thesis in Swing, from ground up. However, for a quick hack, I think Java might be overkill and there’s this Python library I’ve been wanting to play with for so long…

Kivy!

Kivy can handle my UI requirement, and rather beautifully so, if I may say. Now, searching around, there are multiple ways to play an MP3 file with Python. One of them is even a Kivy library! The other libraries I found are mp3play, musicplayer, and PyGame.

But one by one, problems emerged

  • PyGame’s mp3 support is limited.
  • The sample code in musicplayer’s PyPi page looks too complicated. (Yes, since this is just a quick project, I’m in lazy mode.)
  • mp3play only works for Windows XP.

 

And then the Kivy library. It would’ve been sweet to have a single library for all my project’s requirements. But alas…

Kivy has a SoundLoader class which automagically determines the best way to handle the given sound file’s format. I’m not good with design pattern terminology but, the way I see it, it looks like a combination of Factory and Strategy patterns. However, when I test code given in their documentation, I come across the following logs:

[INFO ] Kivy v1.8.0
[INFO ] [Logger ] Record log in /home/chad/.kivy/logs/kivy_15-01-04_6.txt
[DEBUG ] [Audio ] register SoundPygame
[INFO ] [Audio ] Providers: audio_pygame (audio_pygst, audio_sdl ignored)
[WARNING] [Audio ] Unable to find a loader for <test.mp3>

 

The only provider that loaded is PyGame’s. And, as I’ve noted above, PyGame’s mp3 support is shaky. In fact, Kivy’s code as of presstime only allows PyGame on MP3 files if it is running on Android. audio_pygst would’ve done the trick but then, as the logs indicate, it did not load.

So why did it not load? Looking at audio_pygst’s code, it imports the modules gi, pygst, and gst, in that order, respectively surrounded in try-catch statements should there be problems on the import.

I try to load them manually on Python’s shell and get the following result:

chad@galadriel:kivy$ python
Python 2.7.3 (default, Feb 27 2014, 19:58:35) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import gi
>>> import pygst
>>> import gst
/usr/lib/python2.7/dist-packages/gobject/constants.py:24: Warning: g_boxed_type_register_static: assertion `g_type_from_name (name) == 0' failed
  import gobject._gobject
>>>

 

Looking around, this seems to be an issue with gst 0.10.x. The obvious solution would be to upgrade. Unfortunately, gst updates for Ubuntu 12.04LTS seem to have stopped at 0.10.

So there. Much ado achieving nothing, for a quick hack. Maybe, I’ll look into using Java for this project, moving forward. Stay tuned!

Coldplay Fanboy