Author Archives: Grant

About Grant

Grant is currently a lecturer in Engineering at a Prominent North East University in the United Kingdom. Sometimes he is stunned by just how damn good looking he is.... :-) The views contained in these web pages are my personal views and do not represent the views of my employer.

Pivot Tables and Weighted Averages

Pivot Tables are a very useful feature in Spreadsheets, they allow you to do things that might otherwise require code to iterate around on.  My mental model of them is not great though I find there is a great deal of trial and error to get them working for me.

Anyway I recently had to calculate a weighted average and found this web page:

https://excel.tips.net/T002900_Weighted_Averages_in_a_PivotTable.html

very useful.  Essentially you need to add an extra column in your data with an intermediate step of weight*value that you can then sum in the pivot table.

Although the tutorial example there is done in Excel I actual used Libreoffice as despite the fact I now mainly fill out electronic paperwork I have clung onto my Linux box 🙂

Political Protesting – Brexit.

img_-bktk3v

My sign may have been a bit intellectual….

This Saturday I went on my first political protest!  I joined the North East for Europe March on March the 4th.  There were about seven hundred of us in Newcastle to protest about Brexit and the accompanying chaos.  My motivation for going is not just about the referendum of the 23rd of June but the events since then.

I am also absolutely disgusted with the position the government is taking on the issue of EU citizens living in the UK. These are people and to treat them as bargaining chips in a negotiation is simply contemptible. This is not an abstract issue it affects real people right now.  I am unimpressed with the attitude of those who call these people foreigners and act if they matter less as a result.  I don’t call the lovely young lady who married my brother-in-law a foreigner, I call her my sister.

As a remain voter I am also unimpressed by the attitude of the EU in this matter too there are millions of people affected by this issue and I view their lack of effort to resolve this matter rapidly as disappointing to say the least.

Going on a March is a very analogue experience in a digital age.  There were some good speeches, some great speeches, a guitarist or two and quite a lot of shouting.  It was fantastic to be amongst like minded people.  A few interesting takeaways from the speeches which hadn’t occurred to me before:

  • In the UK we have chosen to make employing people here as easy as moving goods across the EU.  So the pressures on our un-skilled and low-skilled labour market are not an automatic result of EU membership and there is a lot more that is under our control than the Brexiters would have you believe.
  • The NHS is already under pressure from a dramatic reduction in recruitment and retention from EU nationals.
  • The current plan for everything to be wrapped up in two years is unrealistic.  This will still be going on at the next election and perhaps the one after that.
  • The current Brexit fallback plan of a deregulated, low tax economy (like Singapore) is something that only works for a small elite who are rich enough not to worry about things like health insurance.

However for me the most striking thing was the very small counter demo, these folk were mainly wearing balaclavas or face masks, saying refugees were not welcome and holding Trump posters.  I was very happy to be counted where I stood….

Plotting 2D Unstructured Data Using Free Software – an update.

A while ago (ten years ago!) I wrote a post called  Plotting 2D Unstructured Data Using Free Software

I thought it might be useful to revisit that post to see how things have moved on.  Nowadays I do most of my plotting in Matplotlib – a library which has matured nicely since 2007, though I get confused quite a lot 😦

In the previous post I looked at GSharp, gri, gnuplot and matplotlib.  I will talk here about only about gnuplot and matplotlib.  Amazingly gnuplot has changed very little – the script I wrote ten years ago works without any modification and produces the same sort of output, i.e. not hugely impressive.  You might be able to get better outputs if you have more skill with gnuplot than me.

gnuplot_contours2

The matplot script however required modification to run.  A user called darksid3 pointed out a much better way of doing in 2010 but I think the matplotlib library has developed somewhat so the code is now very straightforward.  My script is now as follows:

#!/usr/bin/python
"""
Script to plot contour data using Matplotlib
"""
import matplotlib.pyplot as plt
import numpy as np
from scipy.interpolate import griddata

# Read and assign data
data = np.genfromtxt('contourdata.dat',names=['x','y','cp0'])
x = data['x']
y = data['y']
z = data['cp0']

# Do the interpolation
xi = np.linspace(x.min(), x.max(), 500)
yi = np.linspace(y.min(), y.max(), 500)
zi = griddata((x, y), z, (xi[None,:], yi[:,None]), method='cubic')

# Plot using the interpolated data
plt.contour(xi,yi,zi,15,linewidths=0.5,colors='k')
plt.contourf(xi,yi,zi,15,cmap=plt.cm.jet)
plt.colorbar(label="cp0")
plt.xlabel("tang / [m]")
plt.ylabel("radial / [m]")
plt.title("Loss Coefficient CFD Jan 2006 - matplotlib")

plt.show()

You might be able to make it a bit smaller but I favour a longer, readable code over something that is very compact.  The output is pretty straightforward too:

matplotlib_gridata_contours

A better reference can be found in the SciPy cookbook.

So it is nice to see things advancing in the free software world!!

Search Problems with Thunderbird

Recently I have found that Thunderbird search has been a bit annoying.  You will search the database and get results that you want, e.g. the document that Fred sent you two years ago about widgets.  You can see some details about the e-mails – a tantalising glimpse of the header and the attachments but when you click you cannot access the full e-mail.

The solution appears to Rebuild the Global Database: https://support.mozilla.org/t5/Troubleshooting/Rebuilding-the-Global-Database/ta-p/15801

Which I am doing at the moment.  Tools -> Activity Manager allows you to watch the process…..

Update: this works as intended.  I can now find my e-mails in Thunderbird.

The Dreamliner Experience – or why I avoid the 787.

So this summer I had the chance to fly on an almost brand new 787 Dreamliner – having read a number of very complimentary blog posts from business class passengers I was quite excited!

However I rarely fly in business so my experience was that of an economy passenger.  The takeaway point is to avoid the 787 and fly with another aircraft if you possibly can 😦  A lot of the much touted fuel burn improvements seem to be obtained simply at the expense of your comfort!  I am not alone in this view!

So my flight was on a BA 787 from London Heathrow to Montreal (and back).  The way out was a later afternoon flight and way back was an overnight.  There are a number of things wrong with the 787 (as implemented by BA other airlines may differ):

The seats are very narrow.  Both on the way there and the way back you find your shoulders and elbows clashing with your seatmate.  I managed to get an aisle seat each time but on the way back I had to endure some random blokes’ arm digging into my side the entire flight.  It was hot enough so you end up with a sweat patch where his arm touches you – urgh!  Whilst during the flight murdering him seemed a perfectly reasonable course of action, he wasn’t really being antisocial – there is simply not enough room for everyone.  This is a problem as everyone flies the 787 in nine across in economy.  I’ve flow ten across in the back of Emirates 777s loads of times and this felt much worse – I haven’t measured the seats so this is a subjective metric.

The selegroomat pitch is worse than Ryanair!  Again this obviously varies with operator but I had more personal space on a 2 hour trip from Edinburgh to Gdansk than on this transatlantic flight.  See photo left – you don’t get much more legroom with Ryanair but you do get more – a very surprising comparison to me!

The fancy electronic window shades don’t work very well.  So on the way out the cabin crew dimmed the shades but since we were chasing the sunset the sun remained blasting at one side of the plan keeping us in daylight.

The air conditioning system is ineffective.  Since the sun was heating the plane on one side, those of us on the right wereno air vents taking as many of our clothes off as you can get away with and sweating.  On the left side of the plane folk were wrapping blankets around their heads to keep warm.  This might be a function of BAs odd fit which doesn’t seem to fit individual cooling vents (see right).  The average cabin temperature might be fine but the individuals aren’t comfortable – bit of a design failure there!

There is a giant e2015-06-20 05.11.15equipment box in the foot well of every aisle seat.  Again this might be a BA thing but the thing is HUGE – it uses up around 1/3 of the volume.  This means you can’t sit straight in the seat.  This I think runs the entertainment system but is very bulky!

So the thing might be quiet and the cabin humidity might be higher.  I didn’t feel as dried out as I did when I did the Montreal trip in an MD11 the year before – but older aircraft give you much more person space.  Overall it was huge disappointment and I wonder how much the vaunted engineering reduces fuel burn and how much it is simply cramming passengers in like sardines.

I do also think that given that personal space is one of the biggest things being sold on planes that you ought to be able to find out how much space you are allocated easily.    The best way to do this now is via third party websites like Seatguru – which only exist because the airlines don’t provide this information to you.  I don’t see how the market can function without easy comparisons being made.

Anyway to sum up – avoid the 787!

 

Removing more Image Duplicates

Dear Reader,

So whilst working through my cleaned up archive from last week I noticed that I still had duplicate image files!! I have no idea how this happened but in my cleaned up photo directory I have lots files with similar names and file sizes but are not identical…

I should say that they were there before I started my clean up so things are at least getting better.

$ ls -lhrt *2560*
-rw-r--r-- 1 grant grant 838K Apr 9 2009 img_2560.jpg
-rw-r--r-- 1 grant grant 836K Apr 9 2009 IMG_2560.JPG

When you look at the images they are pretty much the same. We can use the identify command from Imagemagick to look at them:

$ identify *2560*
img_2560.jpg JPEG 3264x2448 3264x2448+0+0 8-bit DirectClass 857KB 0.000u 0:00.000
IMG_2560.JPG[1] JPEG 3264x2448 3264x2448+0+0 8-bit DirectClass 855KB 0.000u 0:00.000

The photograph in question is of Venice but actually I have quite a lot of them of various places….IMG_2560

So what to do? After a bit of time with google I found a program called findimagedupes which analyses the images somewhat before deciding they are a match. It takes some time to run but the command I used was:

findimagedupes --fingerprints=imagedupes.dat -R ./ > listofdupes.txt

My listofdupes.txt files was 1614 lines long. This should save a lot of manual deletion! findimagedupes wasn’t perfect though – it matched some things that were not matches so I need to think carefully about what to do with the output… hmm.

Cleanup The Photograph Archive

Dear Reader,

Being a photographer of the “pray and spray” school of composition I have acquired a large number of photographs. I got my first digital camera in around 2004 and now have a collection of nearly 28,000 photographs. Some of them are even in focus.

Due to a number of botched imports many of these are duplicates and they resided in a directory tree that was truly a complete mess. I had three semi-nested directory structures:

  • Pictures/Year/Month/Day with most of the files from 2004 to 2015
  • Pictures/Pictures/Year/Month/Day with some of the files from 2006 to 2012
  • Pictures/Pictures/Photos/Year/Month/Day with more of the files from a random selection of years

Note that Shotwell will detect duplicate files on import – but I had managed to get a lot of files into the archive before I turned this on and I am not sure how well Shotwell deals with the same file with different names img88.jpg and img88-1.jpg or IMG88.JPG for example.

So this weekend I have been sorting this out.    After quite a bit of googling for me this was the step by step process:

  1. Back everything up – in case it all goes wrong.
  2. Set Shotwell to write tags to the files and to watch library location for new files.  This means all the labels will be attached to the pictures rather than Shotwell’s database avoiding potential confusion later.  Setting Shotwell watching the library location allows it to pick up on changes made automatically.   There might be a cleverer way of doing this by fiddling with the database.  My shotwell preferences looked like this:Screenshot from 2015-05-31 22:55:20
  3. For a large archive you need to wait a reasonable amount of time for Shotwell to do its thing…
  4. I then closed Shotwell so it didn’t get upset whilst I moved everything around 🙂
  5. Use fdupes to get rid of the duplicate files. The correction command for this is: fdupes -rdN Pictures/ Note that this removes files without prompting so you should check the output of fdupes -r Pictures/ to make sure that it is about to do what you want. Of course since we carried out step 1 (backup) we can recover from any failure.
  6. Clean up the directory tree using rsync. So basically here I made a new directory called cleanup and moved Pictures/Pictures and Pictures/Pictures/Photos into it. Then rsync -avuP cleanup/Pictures/ Pictures/ and rsync -avuP cleanup/Photos/ Pictures/ was used to sync up the two directory trees. I then deleted the cleanup directory having created a unified structure under Pictures.
  7. Restart Shotwell.  You need to wait quite a bit whilst it works out what you have done to the precious photographs under its guard.  At the bottom left corner you’ll see this chugging away….  Screenshot from 2015-05-31 23:01:15

After all this I ended up with an archive with zero duplicates and a clean directory structure.  The first is objectively useful.  The second is only useful if you interact with the directory tree by hand and since I am using Shotwell is much less important – but I feel better for it nonetheless.