Skip navigation

Category Archives: Bloopers

Coming off from my last post about updating pip inside a virtualenv, I have, finally, realized what I’ve been doing wrong all this time: I use this nifty script called virtualenv burrito to get a virtualenv + virtualenvwrapper set-up.

I’ve been writing an install script for a new project I am working on. Being at such an early phase, the script is scrappy and will only probably work for machines configured exactly as mine. Not that I make a lot of customizations, but still it relies on the existence of virtualenvwrapper too much.

The thing is I could never get my install script to run properly since pip is broken in my virtualenvs. So just yesterday, I finally decided to devote some time to digging into why precisely this is happening.

Long story short, after a few hours digging around the Vogon poetry of bash scripting, I realized that I can’t find separate virtualenv and virtualenvwrapper installs in my system…because I used virtualenv burrito to install them.

So I deleted the .venvburrito directory in my home and rerun virtualenv burrito. And…tadddaaaahhh!

chad@scheherazade:~$ mkvirtualenv fresh
New python executable in /home/chad/.virtualenvs/fresh/bin/python
Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating /home/chad/.virtualenvs/fresh/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/chad/.virtualenvs/fresh/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/chad/.virtualenvs/fresh/bin/preactivate
virtualenvwrapper.user_scripts creating /home/chad/.virtualenvs/fresh/bin/postactivate
virtualenvwrapper.user_scripts creating /home/chad/.virtualenvs/fresh/bin/get_env_details
(fresh) chad@scheherazade:~$ 
(fresh) chad@scheherazade:~$ pip -V
pip 10.0.1 from /home/chad/.venvburrito/lib/python2.7/site-packages/pip-10.0.1-py2.7.egg/pip (python 2.7

What this does not address, however, are my old virtualenvs which are stuck on an outdated pip. But I guess it should be trivial to just delete those virtualenvs and create a new one, this time with an updated pip.

There’s this thing in life, where you have assumptions you neglect to state because you thought (assume) them to be inconsequential, then they turn out to be of great importance. This is all the more true when you work with computers.

I’ve been planning to get a new laptop for quite some time now, maybe at around December. However, certain miscalculations and one heartbreaking failure from the charger of my previous one means that I’m now typing this on a new machine, bought pretty way earlier than expected. And, like all good (or semi-good, depending on your loyalties) geeks do, I set-up the machine to dual-boot Windows 8 (pre-packaged with the hardware) and Linux the first time I had the opportunity (i.e., this long weekend).

It was hell.

Almost like this. Thankfully, only took me 1.5 days.

It seems that my hardships are all courtesy of UEFI/Secure Boot. Maybe there exists an easy way to achieve what I wanted but, judging by the volume of unresolved posts related to my problem scattered on the internet, it is not an obvious way. To anyone who might come across the same problems as me, here’s the complete narrative of how I managed to dual boot my machine.

0 Prologue

I dual-booted on an Acer Aspire-471G laptop. The OSes I want side-by-side is Windows 8 and Ubuntu 12.04 (Precise Pangolin). Since we’re dealing with the BIOS, I think it’s just proper that we get my machine on the record. If you run these steps on another machine, there might be some differences along the way.

Also, it seems that UEFI (and related dual-boot problems) has been around since Windows 7. However, I have never tried dual-booting a Windows 7 machine with Ubuntu—my experiences were all with XP and Vista, and it was all easy. And yes, I also encountered some 7-specific questions along the way.

1 Make sure that your LiveCD is 64-bit

There is no such thing as 32-bit UEFI. You will not be able to boot your machine in UEFI mode from your LiveCD if your LiveCD is  32-bit. While UEFI has support for legacy BIOS (and, from legacy BIOS, allow you to boot from a 32-bit LiveCD) this will result to a rather awkward and possibly-dangerous dual-boot set-up: you’d need to switch your BIOS mode from UEFI to legacy and back, depending on the OS you want to load.

(Blooper disclosure: I almost bricked my BIOS trying to boot from a 32-bit installer in UEFI mode: I ended up installing rEFInd through Windows and renaming stuff from my EFI System Partition, and got all sorts of weird and horrifying errors. The thing is, I just got the installer from one of our sysads at work and have used it previously to dual-boot an old 64-bit machine running XP. I’m glad I realized my mistake early.)

How do you determine then the bitness of your LiveCD installer?

At the root of your LiveCD, there is a file named README.diskdefines. Open that file (right-click->Open with->Notepad, or any other plain text editor). If you are looking at a 32-bit installer, that file will read like this (notice the ARCH variables):

#define DISKNAME  Ubuntu 12.04 LTS "Precise Pangolin" - Release i386
#define TYPE  binary
#define TYPEbinary  1
#define ARCH  i386
#define ARCHi386  1
#define DISKNUM  1
#define DISKNUM1  1
#define TOTALNUM  0
#define TOTALNUM0  1

On the other hand, if on a 64-bit installer, you’ll get:

#define DISKNAME  Ubuntu 12.04.1 LTS "Precise Pangolin" - Release amd64
#define TYPE  binary
#define TYPEbinary  1
#define ARCH  amd64
#define ARCHamd64  1
#define DISKNUM  1
#define DISKNUM1  1
#define TOTALNUM  0
#define TOTALNUM0  1

Also, the 64-bit version will have a directory named “efi” at the root. If you are seeing a 64-bit README.diskdefines and no “efi” directory, there is something wrong with your installer.

2 Get comfortable tweaking your BIOS

Warning: This can be dangerous! Proceed with a well-rested mind.

As you may have guessed earlier, there will be a lot of BIOS dealings involved here. To get to the BIOS set-up, press F2 while your machine boots.1

You need to be comfortable with two things: changing your boot order and adding .efis to the white list (“trusted”) of your machine. The first one is pretty basic but the second one has bit more technical details involved.

(For the first one, aside from being pretty basic, it is oftentimes unnecessary. You just need to make sure that your computer considers the HDD/Windows bootloader later than your optical drive and USB drive for bootable media. This is often the case. ACER ASPIRE-471G USERS, I must note that I was a bit surprised to find out that on this machine, this is not the case.)

White listing .efis involves setting a password and navigating your EFI System Partition (ESP) as well as other possibly-bootable media (like your LiveCD). Just remember two things here:

  • If you mess-up your system and suddenly can’t boot Windows, the .efis which Windows uses are at EFI/Microsoft/Boot and EFI/Boot, not case-sensitive, AFAIK. Might be worthwhile re-white listing the .efis inside these two.
  • If, after setting your boot order, you still can’t boot from your LiveCD installer, you need to white list the LiveCD efi, efi/boot/bootx64.efi

3 Installing Ubuntu

(I’ll be linking this part mostly.)

Now we get to the fun part. If your circumstances are anywhere like mine, you might need to partition your drive first. While you can partition your drive from the Ubuntu installer, in the interest of dual-booting, I recommend that you partition from Windows beforehand2. Here’s how.

Finally, you can proceed with installing Ubuntu. This tutorial pretty much says it all.

Note that the linked tutorial uses an older version of Ubuntu (11.10) which may or may not have caused the existence of the next section of this post which is…

4 PROBLEM!

After successfully installing Ubuntu, I rebooted. Sure the boot-up screen looked a bit different (there was a Unix-y underscore cursor present) but it still loaded directly to Windows 8!

I tried white listing the new EFI/ubuntu/grubx64.efi found in my ESP. However, that resulted to Ubuntu loading directly without even a shadow of GRUB showing up.

I then used the boot-repair utility and got these results. I had to re-white list EFI/ubuntu/grubx64.efi. Indeed, GRUB now shows up but without any option to boot Windows 8.

At this point, it’s been close to 1.5 days since I embarked on this task. I’m glad that xkcd got most of this stuff documented. And I’m also glad that, though I went beyond the 24-hour mark, nothing became a lost cause.

Besides, I’m not a very good swimmer!

In a moment of almost-frustrated defeat, I found zen from an earlier episode in this adventure. It dawned on me: just install rEFInd.

(Yep, my boot-up is now managed by rEFInd. Far from a perfect solution, in my opinion but it does what I wanted it to do. So long for now!)

  1. That’s on this Acer machine. If F2 does not work on yours, other candidates might be F10 and F8. []
  2. In my case, the Ubuntu installer didn’t detect another OS in my machine, didn’t detect Windows 8. I don’t know what would’ve happened if I made a wrong move while partitioning from the Ubuntu installer in this case. Hence, not recommended. []

Note: My thesis is done. However, there are still some things which I find of interest to address here. Due to how our thesis was (mis)handled, I’m only finding the time to post about these things now.

Last time, I told you how we are porting all our Matlab/Octave code to Java. I wrote about a huge difference between Matlab and Octave’s implementation of rgb2gray. However, as it became apparent to me, my issues with rgb2gray are not quite done yet.

First a little background. Our project relies heavily on the Fast Fourier Transform, a numerical algorithm whose products I bet you encounter everyday though you probably are not aware of it. The 2D FFT is built-in with Octave, as expected for a language meant specifically for mathematical processing. The same, however, cannot be said for Java.

It’d be crazy for me to attempt to code it from scratch due to a number of reasons not the least of which is the fact that I don’t really have a solid training with numerical algorithms. So, after browsing through various libraries, we decided to use JTransforms, largely because we can see the source code for ourselves and it is natively in Java, in contrast to, say, FFTW, which has Java bindings but is natively in C. Plus, it does not have power-of-2 constraints, a common trait among various implementations of the FFT.

Now, the thing is, there seems to be quite a considerable offset between the FFT results of Octave1 and JTransforms; the kind of offset which you’d be so tempted to chalk up to floating-point handling differences though some part of your reason tells you it is too large to be so. The offset I was seeing ranged from 2 to 12 integer values.

As I admitted earlier, I don’t have any solid grounding with numerical algorithms. The way I see it, there are a number of possible culprits on why I’m not getting the results I should get. The FFT, after all, is a blanket term for a family of algorithms which all do the same thing, namely, apply the Fourier Transform to a discrete collection of values (i.e., vectors or matrices).

What I’ve Tried (Or, the scientific probable reason on why there is this offset, as discussed by my adviser)

Warning: Some equations ahead. I wouldn’t be delving too much into the equations and you should be able to understand my main point with just some basic algebra. But just so you know.

Consider this tutorial on the FFT.

It defines the 2D FFT as:

F[k,l] = \frac{1}{\sqrt{MN}}\sum_{n=0}^{N-1}\sum_{m=0}^{M-1}f[m,n]e^{-2\pi i(\frac{mk}{M}+\frac{nl}{N})} f[m,n]= \frac{1}{\sqrt{MN}}\sum_{l=0}^{N-1}\sum_{k=0}^{M-1}F[k,l]e^{2\pi i(\frac{mk}{M}+\frac{nl}{N})} 0 \leq m,k \leq M-1, 0 \leq n,l \leq N-1

where M and N is the number of samples in the x and y direction (or, for our particular problem, the dimensions of the image).

While it may seem interesting to find mutual recursion here, what my adviser told me to note is the constant at the beginning of the equations, \frac{1}{\sqrt{MN}}. According to him, depending on the implementation of the FFT we are using (remember what I told you above that Fast Fourier Transform is just a blanket term for a family of algorithms), the constant varies to either \frac{1}{\sqrt{MN}} or \frac{1}{MN}. It’s now just a matter of scaling the samples (the pixel values) by the appropriate constant to get the desired results.

The tutorial I linked to presents the following sample signal to demonstrate the FFT:

Sample Signal

which should evaluate to the following real and imaginary component respectively

FFT real

FFT imaginary

The following code listing directly translates the above example to Octave code:

x_real = zeros(8, 8);
x_real(2,3) = 70;
x_real(2,4) = 80;
x_real(2,5) = 90;
x_real(3,3) = 90;
x_real(3,4) = 100;
x_real(3,5) = 110;
x_real(4,3) = 110;
x_real(4,4) = 120;
x_real(4,5) = 130;
x_real(5,3) = 130;
x_real(5,4) = 140;
x_real(5,5) = 150
x_fft = fft2(x_real);
x_r = real(x_fft)
x_i = imag(x_fft)

Which results to the following output:

x_real =
 
     0     0     0     0     0     0     0     0
     0     0    70    80    90     0     0     0
     0     0    90   100   110     0     0     0
     0     0   110   120   130     0     0     0
     0     0   130   140   150     0     0     0
     0     0     0     0     0     0     0     0
     0     0     0     0     0     0     0     0
     0     0     0     0     0     0     0     0
 
x_r =
 
   1.3200e+03  -7.9113e+02   8.0000e+01  -1.6887e+02   4.4000e+02  -1.6887e+02   8.0000e+01  -7.9113e+02
  -5.0485e+02  -9.0711e+01   2.2142e+02   1.0586e+02  -1.6828e+02   1.2721e+01  -2.6142e+02   6.8527e+02
   1.2000e+02   0.0000e+00  -4.0000e+01  -2.3431e+01   4.0000e+01   0.0000e+00   4.0000e+01  -1.3657e+02
  -3.3515e+02   1.3414e+02   2.1421e+01   5.0711e+01  -1.1172e+02   3.4731e+01  -6.1421e+01   2.6728e+02
   1.2000e+02  -6.8284e+01   0.0000e+00  -1.1716e+01   4.0000e+01  -1.1716e+01   0.0000e+00  -6.8284e+01
  -3.3515e+02   2.6728e+02  -6.1421e+01   3.4731e+01  -1.1172e+02   5.0711e+01   2.1421e+01   1.3414e+02
   1.2000e+02  -1.3657e+02   4.0000e+01   0.0000e+00   4.0000e+01  -2.3431e+01  -4.0000e+01   0.0000e+00
  -5.0485e+02   6.8527e+02  -2.6142e+02   1.2721e+01  -1.6828e+02   1.0586e+02   2.2142e+02  -9.0711e+01
 
x_i =
 
     0.00000  -711.12698   440.00000    88.87302     0.00000   -88.87302  -440.00000   711.12698
  -724.26407   713.55339  -216.56854    55.56349  -241.42136   134.14214   120.00000   158.99495
   120.00000  -136.56854    40.00000     0.00000    40.00000   -23.43146   -40.00000     0.00000
  -124.26407   255.56349  -120.00000    -6.44661   -41.42136    38.99495   103.43146  -105.85786
     0.00000   -68.28427    40.00000    11.71573     0.00000   -11.71573   -40.00000    68.28427
   124.26407   105.85786  -103.43146   -38.99495    41.42136     6.44661   120.00000  -255.56349
  -120.00000    -0.00000    40.00000    23.43146   -40.00000    -0.00000   -40.00000   136.56854
   724.26407  -158.99495  -120.00000  -134.14214   241.42136   -55.56349   216.56854  -713.55339

As evident, the results are very far from what our source says. However, dividing the sample (matrix x_r) by 8 (cf sample size) produces the results as given by our source.

Add the following line before calling fft2

x_real /= 8;

And we get,

 
x_r =
 
   165.00000   -98.89087    10.00000   -21.10913    55.00000   -21.10913    10.00000   -98.89087
   -63.10660   -11.33883    27.67767    13.23223   -21.03553     1.59010   -32.67767    85.65864
    15.00000     0.00000    -5.00000    -2.92893     5.00000     0.00000     5.00000   -17.07107
   -41.89340    16.76777     2.67767     6.33883   -13.96447     4.34136    -7.67767    33.40990
    15.00000    -8.53553     0.00000    -1.46447     5.00000    -1.46447     0.00000    -8.53553
   -41.89340    33.40990    -7.67767     4.34136   -13.96447     6.33883     2.67767    16.76777
    15.00000   -17.07107     5.00000     0.00000     5.00000    -2.92893    -5.00000     0.00000
   -63.10660    85.65864   -32.67767     1.59010   -21.03553    13.23223    27.67767   -11.33883
 
x_i =
 
    0.00000  -88.89087   55.00000   11.10913    0.00000  -11.10913  -55.00000   88.89087
  -90.53301   89.19417  -27.07107    6.94544  -30.17767   16.76777   15.00000   19.87437
   15.00000  -17.07107    5.00000    0.00000    5.00000   -2.92893   -5.00000    0.00000
  -15.53301   31.94544  -15.00000   -0.80583   -5.17767    4.87437   12.92893  -13.23223
    0.00000   -8.53553    5.00000    1.46447    0.00000   -1.46447   -5.00000    8.53553
   15.53301   13.23223  -12.92893   -4.87437    5.17767    0.80583   15.00000  -31.94544
  -15.00000   -0.00000    5.00000    2.92893   -5.00000   -0.00000   -5.00000   17.07107
   90.53301  -19.87437  -15.00000  -16.76777   30.17767   -6.94544   27.07107  -89.19417

Which, at last, agrees with our source.

The Blooper (or, the actual and embarassing reason why JTransforms FFT is not coinciding with Octave FFT)

In my first post about rgb2gray, I mentioned that I was taking the floor of the average of the RGB components of a pixel (or, in Java code, just perform integer division as the resulting precision loss is tantamount to floor). Big mistake. Octave does not merely floor values. It rounds off values.

In code, so there be no misunderstandings,

public static int[][] rgb2gray(BufferedImage bi) {
  int heightLimit = bi.getHeight();
  int widthLimit = bi.getWidth();
  int[][] converted = new int[heightLimit][widthLimit];
 
  for (int height = 0; height < heightLimit; height++) {
    for (int width = 0; width < widthLimit; width++) {
      Color c = new Color(bi.getRGB(width, height) & 0x00ffffff);
      float rgbSum = c.getRed() + c.getGreen() + c.getBlue();
      float ave = (rgbSum / 3f);
      float diff = ave % 1f;
      int iave = diff < 0.5f ? (int) ave : (int) (ave + 1);
      converted[height][width] = iave;
    }
  }
 
  return converted;
}

The moral of this long story? If two supposedly-similar functions are returning different outputs, check that the inputs are exactly the same before jumping to any wild conclusions.

  1. I’d like to note here that, according to the help file, Octave’s FFT is based on FFTW []

*’Cause I have a feeling there’s more to come

I just spent an hour or so figuring out why a long SQL query is returning 0 rows. Long as in,

SELECT books.isbn, title, lastname, firstname, publishername
FROM books
INNER JOIN authored ON books.isbn = authored.isbn
INNER JOIN bookpersons ON authored.personid = bookpersons.personid
INNER JOIN published ON published.isbn = books.isbn
INNER JOIN publishers ON published.publisherid = publishers.publisherid
WHERE title = "Brave New World";

I was already looking into derived relations and subqueries until I realized that tables published and publishers are empty.

I’m calling it a day. So much for populating published and publishers and testing if my query works. Haaaayyyy… OTL