Tuesday, 29 December 2009

A bit of xmas admin support

It started off simple enough: "Can you install Photoshop Elements please?" - A friend got Photoshop Elements for xmas, since it was plugged as an easy, effective way to manage and share photos. (Toshiba laptop, running Vista)

Installing it was easy enough, but when coming to share photos via email it got a bit scary because, he uses Yahoo mail, not a fat email client. When I started looking at it I though okay, I just need to set up Outlook with the mail server settings. Going into the help menu in Yahoo mail gave some pretty clear instructions on how to set Outlook up - but it didn't work. It kept asking for the username and password, which never worked. Searching around the web showed people talking about POP access to Yahoo mail, some suggesting it was a paid feature. But the Yahoo documentation didn't make any mention of needing a paid account.

After wasting too much time on this, I just set up gmail - I knew I could easily access POP and SMTP here. So after creating a gmail account I set the reply to email in Outlook to be the Yahoo account and told gmail to forward any mail to Yahoo. Thus the gmail involvement would be pretty much invisible. I've sent a test email to the gmail account and I'm still waiting for it to arrive at yahoo...

So now I can share photos via email okay. Great. Not particularly easy for the novice user, and how come I can't just tell PSE to use a web mail provider? AND, it looks like PSE maintains its OWN list of contacts. Not related to the Yahoo or Outlook contact lists. Jeeezz.

Next, to upgrade AVG virus checker. I was stunned to find AVG Free wouldn't let me switch the Firefox search box provider to Google. Apparently it switches your default to yahoo, and denies you when you try to change it. I think its all configurable, but thats a crazy thing to deny you by default. Crazy.

So, I download the 1MB installer. Which then proceeds to download who knows how many MBs - but it takes a LONG time over a mobile internet connection (Three). Then the install starts, and promptly stops telling me to uninstall Microsoft Live OneCare - which I do, which means a reboot, and then when I restart the installer, it downloads everything again. This time, I did not install the link checker hoping to avoid the "cripple my computer" feature.

The last thing to do - disable the touch pad when the usb mouse is plugged in - was easy (via the control panel mouse settings).

3 Hours later I'm done. But we've got a long way to go baby! This was way too hard. I hope I'm totally wrong - Perhaps there are easier ways, but they weren't obvious and I don't have another 8 hours to figure it out. If I was a billionaire, I know what I'd be doing - sorting this sh*t out!

Wednesday, 25 November 2009

A simple task queue

I'm working on a little sample framework - really only to keep my sanity and practice my chosen craft - that allows you to string together tasks in a pipeline for processing. To exercise that framework - to flush out the pros and cons of the implementation - I'm writing a sample application.

The basic idea is that each task is interested in an event. That event could be the arrival of a file, or the completion of another task etc. To string tasks together, I've created a simple database queue system - as a task completes, it writes to the queue and then other tasks which are interested will see that event and then begin. Tasks complete either successfully (by returning no errors) or unsuccessfully (by returning more than one error).

An important concept is to be able to add (or remove) tasks from the pipeline programmatically and without changing the database schema. So, quite simply, when a task completes, that event is written to the queue - but how do we find which events another task is interested in? And how do we do this in a way that won't take longer over time?

So, lets assume Job2 is interested in the successful completion of Job1. We need to find all successful Job1 events which haven't already been processed by Job2 (successfully or otherwise):

[sourcecode lang='sql']
SELECT * FROM QUEUE where job='job1' and result='SUCCEEDED' and task not in (SELECT task FROM QUEUE where job='job2' and result is not null)

I'm no SQL ninja, so this is my first simple solution. It works, but it doesn't scale. With a thousand rows, it completes pretty quickly, but with tens of thousands of rows it takes way too long (nearly 8 minutes with 40000 rows running on my Dell Inspiron laptop).

Pragmatically, we can restrict the query based on time - we don't need to search the whole table, just the more recent events:

[sourcecode lang='sql']
SELECT * FROM QUEUE where tstamp >= ? and job='job1' and result='SUCCEEDED' and task not in (SELECT task FROM QUEUE where tstamp >= ? and job='job2' and result is not null)

You could easily just look over the events in the last hour if your tasks are short running and you get less than a thousand events an hour - it'd probably perform okay. To give an indication of performance I've written a little groovy script to show some basic trends - included below.

paul@paul-laptop:~$ groovy sqltest
2005 rows took: 4
4005 rows took: 5
6005 rows took: 3
8005 rows took: 3
10005 rows took: 2
12005 rows took: 3
14005 rows took: 3
16005 rows took: 4
18005 rows took: 5
20005 rows took: 3

Notice though, that this is a great example of where indexes really help - without the index times get larger as the rowcount gets larger:

paul@paul-laptop:~$ groovy sqltest
2005 rows took: 13
4005 rows took: 34
6005 rows took: 123
8005 rows took: 134
10005 rows took: 229
12005 rows took: 237
14005 rows took: 330
16005 rows took: 342
18005 rows took: 428
20005 rows took: 442

Its a pity I have to worry about the time restriction, but it does make the solution workable and at least in my intended application, appropriate. I was hoping to use the SQL MINUS function, but it appears MYSQL doesn't support it.

Here's the groovy script I used to generate the results. The difference between using the time restriction and the index is so dramatic and obvious, it is a great example.

[sourcecode lang='java']
import groovy.sql.Sql
import groovy.grape.Grape

Grape.grab(group:'mysql', module:'mysql-connector-java', version:'5.1.10', classLoader: this.class.classLoader.rootLoader)


def go() {
def sql = Sql.newInstance("jdbc:mysql://localhost:3306/spike", "spike","password", "com.mysql.jdbc.Driver")
try {
sql.execute("drop table QUEUE")
} catch(Exception e){}

sql.execute("CREATE TABLE QUEUE (id INTEGER NOT NULL,job varchar (20) NOT NULL,task varchar (20) NOT NULL,result VARCHAR (20) NOT NULL, tstamp timestamp, PRIMARY KEY (id)) ENGINE = MyISAM")
sql.execute("create index idx1 on QUEUE(tstamp, job, result)")

(1..10).each() {
def d = createHistory(sql,1000*it)
findJobsToProcess(sql, d)

def findJobsToProcess(sql, d) {
def rowcount = 0
sql.eachRow("select count(*) from QUEUE") { row ->
rowcount = row[0]

long start, end

start = System.currentTimeMillis()
// now find the 'job1' items that haven't been processed by 'job2'
sql.eachRow("SELECT * FROM QUEUE where tstamp >= ? and job='job1' and result=? and task not in (SELECT task FROM QUEUE where tstamp >= ? and job='job2' and result is not null) order by tstamp asc", [d,STATUS.SUCCEEDED.name(),d]) {
// println "${it.id} ${it.job} ${it.task} ${it.result} ${it.tstamp}"
end = System.currentTimeMillis()
println "${rowcount} rows took: ${end-start}"


def createHistory(sql, count) {
sql.execute("truncate table QUEUE")
int id = 1
int task = 1
// create a "history" of previous jobs
STATUS.each() { status ->
(1..count/2).each() {
sql.execute("insert into QUEUE (id,job,task,result,tstamp) values (?,?,?,?,?)", [id++,'job1',task.toString(),status.name(), new Date()])
sql.execute("insert into QUEUE (id,job,task,result,tstamp) values (?,?,?,?,?)", [id++,'job2',task.toString(),status.name(), new Date()])
def d = new Date()
// insert some successful 'job1' items which 'job2' hasn't processed
(1..5).each() {
sql.execute("insert into QUEUE (id,job,task,result,tstamp) values (?,?,?,?,?)", [id++,'job1',task.toString(),STATUS.SUCCEEDED.name(), new Date()])
return d



In running this test, I used:

  • Dell Inspiron 1525, Intel(R) Core(TM)2 Duo Processor T5550, 1.83 GHz, 2MB Cache, 667 MHz FSB, 2GB (2 X 1024MB) 667MHz Dual Channel DDR2 SDRAM, 160GB 7200RPM Performance Hard Drive

  • Ubuntu 9.10 32 bit desktop edition

  • Groovy Version: 1.6.5

  • JVM: 1.6.0_03

  • mysql Ver 14.14 Distrib 5.1.37, for debian-linux-gnu (i486)

Monday, 16 November 2009

Rapache on Ubuntu 9.10

I've just now stumbled across Rapache, a useful GUI tool that makes configuring apache easy. I found it by accident in the Ubuntu Software Center, but unfortunately it would freeze while trying to add a new domain. I searched the web for answers, and found a bug report.

This didn't specifically reference Ubuntu 9.10 (rather, 9.04) and the file that needed to be patched didn't exist in the given location.

I found it easily enough:

paul@paul-laptop:~$ sudo find / -name RapacheGui.py[sudo] password for paul:

I edited the last one (/usr/share/pyshared/RapacheGtk/RapacheGui.py) as documented in comment 23 to add the following at line 79:

if not Shell.command.ask_password(): sys.exit(1)

Note, this line MUST be preceeded by 8 spaces - indentation is important in Python.

Now, rapache would prompt for the system password and then close!

I've got it working now, by starting it with sudo:

sudo rapache

Everything seems to work. I've added a domain, turned on the include module, and added "AddOutputFilter INCLUDES .html" to the virtual host definition - all very quickly using rapache.

So, server side includes now work on my new virtual host - which is what I started out wanting to do before going off on this tangent.

Saturday, 7 November 2009

Tagging with Appengine DataStore

I've been looking into how to implement a tagging mechanism with Appengine (Python). By using a StringListProperty, you can associate a list of tags with an entity. The model would look something like this:

from google.appengine.ext import db

class Sample(db.Model):
name = db.StringProperty(required=True)
tags = db.StringListProperty()

Now, assuming we want to find the entities tagged with a given word, we can use a query like this:

q = db.GqlQuery("SELECT * FROM Sample where tags = :1",'z')

If we want to find all entities that have ANY of these tags (OR) we can query with GqlQuery or filter:

q = db.GqlQuery("SELECT * FROM Sample where tags in :1",['a','z'])

q = domain.Sample.all()
q.filter("tags in ", ['b','d'])

When we want to find entities with ALL of the given tags:

q = db.GqlQuery("SELECT * FROM Sample where tags = :1 and tags = :2",'b','c')
q = domain.Sample.all()
q.filter("tags = ", 'b')
q.filter("tags = ", 'c')

Too easy!

If you have an appengine project, you can try this code out by adding the entity (Sample - listed above) to your domain model and run the following in your development console (http://localhost:8080/_ah/admin/interactive):

import domain


def display(r):
for r in results:
print r.name +" tags = "+' '.join(r.tags)

q = db.GqlQuery("SELECT * FROM Sample where tags = :1",'z')
results = q.fetch(5)

q = db.GqlQuery("SELECT * FROM Sample where tags in :1",['a','z'])
results = q.fetch(5)

q = domain.Sample.all()
q.filter("tags in ", ['a','z'])
results = q.fetch(5)

q = db.GqlQuery("SELECT * FROM Sample where tags = :1 and tags = :2",'b','c')
results = q.fetch(5)

q = domain.Sample.all()
q.filter("tags = ", 'b')
q.filter("tags = ", 'c')

results = q.fetch(5)

This should produce the following result:

t3 tags = x Y z
t1 tags = a b c
t3 tags = x Y z
t1 tags = a b c
t3 tags = x Y z
t1 tags = a b c
t2 tags = b c d
t1 tags = a b c
t2 tags = b c d

So, it's easier than I thought. But I'm not experienced with DataStore yet, and I don't know if there are inherit limitations with this approach - remember, there are limits on the way you retrieve data.

Tuesday, 27 October 2009

No module named _multiprocessing

I'm getting back into appengine at the moment, and I came across this error:

No module named _multiprocessing

A google search turned up this error report:


which lead to this post:


The problem was of-course, that I was using Python 2.6 instead of 2.5.

I love Linux, the solution is trivial:
The workaround for this is to install Python 2.5 on your machine. It will be then be accessible by using python2.5 on the command line. Then open thirdparty/google_appengine/dev_appserver.py in your favorite text editor and replace the first line,

#!/usr/bin/env python


#!/usr/bin/env python2.5

Note, I already had Python 2.5 installed via:

sudo apt-get install python2.5

Now, I'll promptly get back to figuring out how best to use DataStore to implement my solution!

Sunday, 11 October 2009

Amazon web services via Grails

I just recently noticed that Amazon have changed the way you invoke their Product Advertising API - you now need to sign requests as described in the Example REST Requests.

I obviously haven't been reading my emails, since the first clue came when I saw:

<Error><Code>MissingParameter</Code><Message>The request must contain the parameter Signature.</Message></Error>

Luckily, they have a java example demonstrating how to make an API request.

If you extract the source from this sample into your <GRAILS_APP>/src/java directory, then you can invoke the amazon api with code like:

[sourcecode language="java"]
import org.codehaus.groovy.grails.commons.ConfigurationHolder
import com.amazon.advertising.api.sample.*

class AmazonService {
private static final String ENDPOINT = "ecs.amazonaws.com";
private static final String AWS_SECRET_KEY = "put your secret key here";
private static final String AWS_ACCESS_KEY_ID = "put your access key here";

boolean transactional = false

* Call amazon lookup api.
* Returns xml document.
def getData(String asin) {
SignedRequestsHelper helper = SignedRequestsHelper.getInstance(ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_KEY);

String requestUrl = null;
String title = null;

Map params = new HashMap();
params.put("Service", "AWSECommerceService");
params.put("Version", "2009-03-31");
params.put("Operation", "ItemLookup");
params.put("ItemId", asin);
params.put("ResponseGroup", "Small,Medium");

requestUrl = helper.sign(params);

def xml = new URL(requestUrl).text
return new XmlSlurper().parseText(xml)

You could, of-course, define your access keys in the Config.groovy class if you prefer.

(To find your identifiers, see http://docs.amazonwebservices.com/AWSECommerceService/latest/DG/index.html?ViewingCredentials.html)

Saturday, 10 October 2009

Locked subversion working copy

I don't know how it happened, but my local working copy got locked. This meant that every time I tried to commit, I got a 'working copy is locked' error.

The solution was easy, after I found a reference in the FAQ:
svn cleanup <directory>

Default open action for Ubuntu

Something happened on my Ubuntu install sometime which meant that double clicking on an HTML file resulted in the file being opened by gedit. Not really what I wanted. I started looking through the GNOME system menu (Preferences and Administration) to find how to change the default action - but that turned out to be the wrong place.

In Nautilus, simply right click on the file and select 'Properties'. From there, go to the 'Open with' tab and select the appropriate application.

Friday, 21 August 2009

The problem with Digital TV

The problem with Australian Digital TV or at least, using a media center in Australia is:

  • The standard def and high def channels are almost the same, but not quite. The high def channels have subtly different programs such as special documentaries.

  • The EIT is a mess - spelling mistakes, program name changes (series premire, season finale, double episode etc), and not starting/finishing on time.

The consequence of (1) is that you can't just go and say 'record all' - you've then got to go edit that rule to say 'only on this channel', otherwise you'll end up with 2 recordings of the same program, one standard def, one high def. You wouldn't actually want to delete one of the channels, because the differences can be interesting.

The EIT issue is more serious - instead of being able to say 'record all occurances of "Program 1" only on this channel' you need to change this to not be so exact on the program name and use "%Program 1%" so that you catch episodes named 'Program 1 - season finale'. I've seen some programs use the subtitle in the name - so EVERY episode has a different name - Come on! This is what the description is for - instead of:
Name: 'Program 1 - the one where Mr X falls over'

it's supposed to be:
Name: 'Program 1'
Description: 'The one where Mr X falls over'

The starting and finishing on time may be the most frustrating though, since right now I have MythTV configured to start recording 10 minutes early and finish recording 10 minutes after the scheduled finish time. And I still miss the end of some programs!

I can understand that these issues may be planned incompetence rather than innocent mistakes to force you to watch live TV and to pay more attention to the TV guide, but I just don't have time for it.

Once you get a media center it changes how you watch TV - if I have downtime, I can just browse the recordings to find something I'm in the mood for.

Wednesday, 19 August 2009

The week in review - 2009-32

SpringSource Roo
I just found out this week about SpringSource Roo - a rapid development tools that promises:
Working applications within 10 minutes of finishing the download

I haven't used it yet, but at first glance it looks like a Grails/Rails type tool but being plain Java based as opposed to Groovy or Ruby based. This is a great move, since enterprises just don't seem to be able to get comfortable with dynamic languages - I mean, why continue worry about every little nut and bolt when Grails takes care of all that detail - Roo may just be the ticket for those too afraid to make the jump to Grails.

SpringSource STS
I've been using Netbeans 6.7 for Grails development these days (with what little time I have), but I'm thinking about trying out the latest SpringSource Tool Suite (STS) - I'm a big fan of everything SpringSource does, and I'm keen to see what their Groovy support in Eclipse is like.

Replacing a DVD drive in Mythbuntu
When I originally set up Mythbuntu media center, I had a CD-RW drive and DVD-RW drive (both IDE). The DVD drive recently stopped working (could never eject) so I bought a new SATA DVD+RW drive. When I went to install it I found I'd run out of SATA power connectors (I've got 4 disk drives and apparently only 4 power connectors). No problem, I just had to buy a Molex-to-SATA power adapter cable - this plugs in to the power sockets I was using for the IDE drives on one end, and presents a SATA power connector on the other (costs approx AUD$5.50).

However, after I powered it up, it was registered as /dev/dvd3. This didn't work with MythTV because it expects to use /dev/dvd for playing disks. Since I've removed both the CD and DVD drive I *want* it to be /dvd/dvd - I mucked around with symbolic links for a while, but had no success. I don't know anything about Linux devices, but I managed to stumble my way through, and found the answer in /etc/udev/rules.d/70-persistent-cd.rules. This file seems to map each device to a symlink, so all I did was comment out all of the listed devices and rebooted - after the restart, the dvd was correctly registered at /dev/dvd. Apparently it IS easy when you know how.

Unfortunately when I went to test DVD playback, I must have managed to pick up the ONE disk in my collection that was just never going to work. I spent the rest of the night trying to find out why I couldn't play it, suspecting it was a CCS/encryption problem only to try another disk several days later without any problems - I *should* know better by now.

Restoring Vista
I recently had to reinstall a friends Vista powered laptop. It was a 1 month old Toshiba, and it just wouldn't start. I guessed a reinstall was in order, so I booted an Ubuntu Live CD so I could back up any personal data. These days you don't get the physical install disks any more - there is a hidden partition on the hard disk which can be used to restore the disk image. It seems you can access this restore program by holding down 0 (zero) while powering on. Restoring the disk image worked, although it took an incredible amount of time to complete. After it finished, I noticed that c: had 43GB used. I don't know what Toshiba had there besides Vista, but 43GB for the base install is incredible. Last time I installed Ubuntu 9.04, it used less than 2GB! Laptop disks are still around the 250GB mark so after a 43GB operating system you've lost 20% of it. I assume the hidden restore partition is also eating into this disk space?

Friday, 24 July 2009

Sharing data between applications

Sometimes applications need access to data from another application. Here is one approach you could take:

  1. Define a view in the database that owns the data - this view should expose the data that the external application needs

  2. Grant select permissions so that the external application can select from the view

  3. Create a synonym in the external applications database for the view so that it appears as a local resource

This has the advantage that the owner of the data defines what is exposed. Handy if the data is sensitive in any way.

This solution also assumes that the external application only wants to read the data. If it wanted to write to the database, then the application that owns the data should probably publish an API (eg web services et al).

Saturday, 11 July 2009

Adding items to Gnome menus

I've always wondered how to add items to Gnome menus. Well, it turns out it is relatively easy - as you would expect - and its documented here.

It's as simple as creating a text file in the right place - refreshing easy compared to the binary shortcuts in Windows.

When I was working on ToolInstaller, I added support for Windows shortcuts by using the OrangeVolt ant tasks library.

Now that I have the missing piece of the puzzle, ToolInstaller could be easily modified to support Linux (GNOME).

ToolInstaller was a project I started years ago - it was inspired by how easy it was to install software on linux (i.e. apt-get, yum) by using a repository of software, combined with the simplicity of programs which can just be unzipped.

Friday, 5 June 2009

No Scope registered for scope request

Recently I was working with some code which defined beans as request scope. Junit tests failed with the following exception while trying to initialize the application context:

Caused by: org.springframework.beans.factory.BeanCreationException: Could not autowire field: ...; nested exception is java.lang.IllegalStateException: No Scope registered for scope 'request'

I found the solution by searching around (unfortunately I didn't record the sources). Basically, you can register the required scopes as shown below:

[sourcecode language='java']
import org.springframework.mock.web.MockHttpServletRequest;
import org.springframework.web.context.request.RequestContextHolder;
import org.springframework.web.context.request.RequestScope;
import org.springframework.web.context.request.ServletRequestAttributes;
import org.springframework.web.context.request.SessionScope;

public class SampleTest extends AbstractDependencyInjectionSpringContextTests {

protected void prepareTestInstance() throws Exception {
context.getBeanFactory().registerScope("session", new SessionScope());
context.getBeanFactory().registerScope("request", new RequestScope());
MockHttpServletRequest request = new MockHttpServletRequest();
attributes = new ServletRequestAttributes(request);


Friday, 29 May 2009

Trinidad and text wrapping

While using Trinidad 1.0.1 I came across a problem where text did not wrap when displaying read only text fields. This meant that everything was pushed off the right hand side of the page. I had to change the fields from:
<tr:inputText label="#{msgs['message1']}" styleClass="labelRight"
id="mybean" columns="10"
<tr:panelLabelAndMessage label="#{msgs['message1']}">
<tr:outputText value="#{mybean.value}" /> </tr:panelLabelAndMessage>
and added the following to my style sheet:
.af_panelFormLayout_content-cell {
white-space: normal;
Note outputText instead of inputText. It was tricky to get it working in FF and IE at the same time.

Sunday, 17 May 2009

Virtual machines with VirtualBox

I've found VirtualBox to be very easy to create virtual machines - it's free, open source, and available on Windows, Linux, Macintosh and OpenSolaris. One of the features I particularly like is being able to create a virtual disk that grows as needed (dynamically expanding image). You can also mount an iso image for the CD/DVD drive - which is exactly what I want to do when installing the operating system on my new virtual machine.

Installing on Ubuntu is as easy as:

sudo apt-get install virtualbox

Now, from the Applications/Accessories/VirtualBox OSE menu item, you can start VirtualBox and create virtual machines.

Several things I need to remember after I've created a new machine:

  1. Under "Settings/General/Advanced" select "Enable PAE/NX" to avoid the "This kernel requires the following features not present on the CPU: 0:6 Unable to boot - please use a kernel appropriate for your CPU."

  2. Under "Settings/Network/Adapter 1" select "Attached to: Host Interface" to get networking going - this get me network access as I would expect

Wednesday, 13 May 2009

Finding constraints in Oracle

It can be quite frustrating when you get exceptions like those below - constraint violations where the constraint has a system generated name which means nothing to you:

2009-03-23 23:30:43 ERROR [AbstractFlushingEventListener.performExecutions] Could not synchronize database state with session
org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update
Caused by: java.sql.BatchUpdateException: ORA-00001: unique constraint (SAMPLE.SYS_C00123456) violated

The easiest way to find out which table this constraint is associated with is to issue a statement as shown:
select constraint_name, table_name from user_constraints where constraint_name='SYS_C00123456';

Now you know the table involved, you can easily find out the exact nature of the constraint in various ways - my tool of choice is SqlDeveloper - a GUI interface to Oracle (and other) database.

Tuesday, 5 May 2009

Just for a bit of fun... whohasmorefollowers.com

Recently I jumped on the Twitter bandwagon and created whohasmorefollowers.com - a simple service that lets you compare the number of followers for 2 Twitter users.

This weekend I added a game feature to it - the player is presented with 2 random twitter users (from the top 50) and is challenged to guess which user has the most followers. It makes you think, and the results can surprise you!

Why build such a thing? Well, just for a bit of fun really. But it was also a good excuse to familiarise myself with:

  1. jquery

  2. jquery plugins (jquery.template)

  3. blueprint css framework

Another added benefit was with having such a simple application was that the whole exercise fitted into a very short timebox. The exposure to these frameworks made it worthwhile doing, and it reminded me how painful IE can be.

If you're interested, play the game and tweet your score!

Saturday, 18 April 2009

Redirecting javathinking.com to www.javathinking.com

I thought it would be a good idea to redirect requests to http://javathinking.com to http://www.javathinking.com and I found out how to do that easily here.

Basically I just needed to add the apache directives:
RewriteEngine on
RewriteCond %{HTTP_HOST} ^javathinking.com
RewriteRule ^(.*)$ http://www.javathinking.com$1 [R=permanent,L]

In ISPConfig this can be done easily through the administration console rather than editing the apache config files.

Firebug shows me that I am now getting a 301 redirect when accessing http://javathinking.com.

Tuesday, 14 April 2009

cp with force overwrite

I was recently trying to copy a directory over an existing one and kept getting prompted - asking if I wanted to overwrite existing files - even though I was using the force option:
cp -Rf dir1 dir2
I found the answer here - it was caused by an alias forcing the interactive mode.

Quick solution is to bypass the alias by an absolute reference to cp:
/bin/cp -Rf dir1 dir2

Wednesday, 1 April 2009

Error executing script War: No such property: stagingDir for class:_Events

While trying out my new Morph Grails hosting account, I installed the morph-deploy plugin and then promptly discovered that 'grails war' failed with

Error executing script War: No such property: stagingDir for class: _Events

It turns out this is a known problem with Grails 1.1 and the solution is to delete the _Events.groovy file in

[your home dir]/.grails/1.1/projects/[your application]/plugins/morph-deploy-0.1/scripts

Modifying web.xml in Grails

I'm trying out Grails application hosting at Mor.ph with a free developer account. To put up a sample application, I wanted to be able to protect the site with basic authentication so that it won't be accidentally found and played with. So, following the Mor.ph instructions, I needed to modify web.xml to add security constraints and a login mechanism.

As with most things Grails, the solution was easy - simply install the templates and modify the web.xml template.

But, at the moment I have to uncomment it in order to get it running locally - what I really need is a way to only put this modification in when the environment is production...

Sunday, 22 March 2009

Playlists with MythTV

I've been wanting a way to use playlists on my media center either from within MythTV or with any other player. The reason is I have a bunch of short videos (music and documentaries) that I'd like to be able to play consecutively.

It turns out to be a case of just RTFM. You can create playlists for MPlayer from directory listings, and then associate playlists with a particular mplayer command.

To generate the playlist from all files in a directory, I use the following:

dir * | sed 's/\\//g' > all_videos.pls

The sed part of this command removes the \ from escaped spaces... i.e. a file 'Hello world' is output from the dir command as 'Hello\ world' - the space is escaped. We need to generate a file without these escaped spaces for it to work properly.

I use the following mplayer command associated with the pls extension:
mplayer -shuffle -fs -zoom -quiet -vo xv -playlist %s

Saturday, 21 March 2009

Hey! Where'd my swap go?

I've just noticed that I've got no swap space! (Dell 1525 running Ubuntu 8.10).

I first installed my laptop with Ubuntu 8.04, and later upgraded to 8.10. After the upgrade I noticed that hibernate no longer worked, and I think (from memory) I found an error message saying something about not enough swap space. I briefly thought about trying to find out how to add more swap (I already had a 4G partition defined, and I've only got 2G ram) but then I swiftly moved on to other more pressing issues.

Well, just recently I fired up System Monitor to check on a process, and notice in the bottom right corner it said there was no swap space. Firing up 'top' confirmed this - my system had zero swap!

I could see that I had a swap partition defined:
paul@dell1525:~$ sudo blkid
/dev/sda8: TYPE="swap" UUID="4bd53cb9-611c-4e57-a9df-4754d6bcdd65"

But the corresponding entry in /etc/fstab had a different entry:.
# /dev/sda8
UUID=e6eb6c47-599c-4612-ac8a-287279ee438b none swap sw 0 0

The UUIDs didn't match! Thus no swap. Correcting the fstab file and rebooting resolved the problem, but I don't know what caused it.

Right now, top shows I've got my 4G swap space being used, and hibernate has successfully hibernated and resumed. Perhaps now this laptop will perform better too?

As for adding more swap space, I don't think I need to do that right now - but here is a good article for future reference.

Wednesday, 18 March 2009

Grails Openid plugin and Xerces

I just created a grails 1.1 application, and installed the openid plugin. When starting the application, I got the following exception:

2009-03-17 22:25:41,090 [main] ERROR context.ContextLoader - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'consumerManager': Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [org.openid4java.consumer.ConsumerManager]: Constructor threw exception; nested exception is java.lang.NoClassDefFoundError: org/apache/xerces/parsers/DOMParser
at java.security.AccessController.doPrivileged(Native Method)
Caused by: java.lang.NoClassDefFoundError: org/apache/xerces/parsers/DOMParser
at org.openid4java.discovery.Discovery.(Discovery.java:47)
at org.openid4java.consumer.ConsumerManager.(ConsumerManager.java:51)
... 21 more

I worked around this by dropping xercesImpl-2.6.2.jar in the lib directory of my grails app. This seems to work - I've successfully logged in via my yahoo openid account.

I haven't looked any further into this - time is short already, but I hope this helps you if you stumble across this problem.

Set default browser in Ubuntu

To set the default browser, use update-alternatives as shown below:

paul@dell1525:~$ sudo update-alternatives --config x-www-browser

There are 2 alternatives which provide `x-www-browser'.

Selection Alternative
1 /usr/bin/firefox-3.0
*+ 2 /usr/bin/seamonkey

Press enter to keep the default[*], or type selection number: 1
Using '/usr/bin/firefox-3.0' to provide 'x-www-browser'.

There is a good write-up of the update-alternatives system here.

Tuesday, 17 March 2009

Podcatching solution

I'm getting closer to a podcatching solution with gpodder. Gpodder is a simple and easy to use podcatcher - and importantly, it can be invoked from the command line in a non-interactive mode. This is important because of my internet plan - I get twice as much download quota on off-peak times as I do on on-peak. So, I can shedule a cron job to fire gpodder up at 1am and download the latest podcasts. Great, because I'm not going to consume precious on-peak bandwidth.

Now that I've got the latest podcasts handy, I can listen to them on my media center using Amarok. This isn't the only option, but it works for me because the interface lets me 'group by album' - which is conveniently groups most podcasts - and display those added today/1 week/1 month/...

Amarok also lets me configure Xine to output to SPDIF - important, because my media center is only connected via SPDIF to my surround sound receiver. Without this option, I wouldn't be able to hear anything.

A little frustrating is that gpodder saves all of its files using hashes instead of human readable filenames (it looks like there may be a solution to this in the next release). This doesn't really matter if you use a media player that reads the mp3 tags - as most do, including Amarok - but I've been surprised by several podcasts that have no mp3 tags at all:

paul@dell1525:~$ wget http://www.stanford.edu/group/edcorner/uploads/podcast/holmes090304.mp3
2009-03-16 21:41:29 (552 KB/s) - `holmes090304.mp3' saved [27288970/27288970]
paul@dell1525:~$ id3ed -i holmes090304.mp3
holmes090304.mp3: (no tag)
paul@dell1525:~$ id3v2 -l holmes090304.mp3
holmes090304.mp3: No ID3 tag

paul@dell1525:~$ wget http://files.libertyfund.org/econtalk/y2009/Waleswikipedia.mp3
2009-03-16 21:44:35 (530 KB/s) - `Waleswikipedia.mp3' saved [19987016/19987016]
paul@dell1525:~$ id3ed -i Waleswikipedia.mp3
Waleswikipedia.mp3: (no tag)
paul@dell1525:~$ id3v2 -l Waleswikipedia.mp3
Waleswikipedia.mp3: No ID3 tag

Compare this to a podcast with mp3 tags:

paul@dell1525:~$ id3v2 -l grails_podcast_episode_63.mp3
id3v2 tag info for Podcasts/Grails Podcast/grails_podcast_episode_63.mp3:
TT2 (Title/songname/content description): Grails Podcast Episode 63: Newscast for August 17, 2008
TP1 (Lead performer(s)/Soloist(s)): Glen Smith & Sven Haiges
TP2 (Band/orchestra/accompaniment): Glen Smith & Sven Haiges
TCM (Composer): Glen Smith & Sven Haiges
TAL (Album/Movie/Show title): Grails Podcast
TYE (Year): 2008
TBP (BPM (beats per minute)): 30720
TCO (Content type): Podcast (255)
COM (Comments): (iTunPGAP)[eng]: 0
TEN (Encoded by): iTunes v7.7.1
COM (Comments): (iTunNORM)[eng]: 000001C2 000001C2 00006F76 00006F76 0009BCA7 0009BCA7 0000786C 0000786C 0008439C 0008439C
COM (Comments): (iTunSMPB)[eng]: 00000000 00000210 00000B34 00000000048F933C 00000000 01A793ED 00000000 00000000 00000000 00000000 00000000 00000000
PIC (Attached picture): ()[JPG, 0]: , 17185 bytes

Another feature of gpodder that I appreciate is synchronization to media device OR filesystem. The filesystem synchronization means you can specify a directory to copy to and the filename format. This suits me well (although this doesn't seem to copy those files without mp3 tags).

Monday, 16 March 2009

Extra Ubuntu Repositories

A friend put me on to this blog post about extra repositories for Ubuntu. This is a great resource which highlights some great (and possibly essential) software that you should know about if you are running Ubuntu (and possibly any linux flavour).

Have a look, you may find some software listed there of interest to you. I didn't know about some of the items listed (eg GNOME Do) so its been a great resource for me.

Getting rid of Root email

I've got a Linux VPS through RimuHosting - I'm no system admin, and the root account gets a pile of email spam. This takes up disk space so I try to regularly delete it. The best way I've found is to us Mutt - a console based email reader.

Firing up Mutt as root displays the thousands of spam mails recieved. Pressing 'D' lets you delete messages that match a pattern. So I can specify the pattern 'a' and that matches almost all of the messages. When quitting the application, it deletes the marked messages.

So, this is a good solution for reading or deleting many emails without downloading them - if you have ssh access.

To avoid having to do this, can anyone tell me how to stop receiving these emails?

Sunday, 15 March 2009

Removing svn files

I recently needed to clean up and remove all of the svn files in a directory structure. I found the answer here, a simple linux command line to recursively delete .svn folders:

find . -name ".svn" -print0 | xargs -0 rm -Rf

Saturday, 14 March 2009

Over typing -Dskip.junit=true

I generally prefer Maven2, but I'm currently on a project using a custom ant build script - and I'm over having to type -Dskip.junit=true when I just want to generate a quick jar without running the tests.

To easily switch off tests, I modified ant.bat so that it looks for a command line parameter 'st' and if found, it will substitute it with -Dskip.junit=true.

In Ant 1.7.0, you just need to change line 68 in block ':setupArgs' from



if ""%1""==""st"" (set ANT_CMD_LINE_ARGS=%ANT_CMD_LINE_ARGS% -Dskip.junit=true) else (set ANT_CMD_LINE_ARGS=%ANT_CMD_LINE_ARGS% %1)

So, the whole block reads (from line 65):

if ""%1""=="""" goto doneStart
if ""%1""==""-noclasspath"" goto clearclasspath
if ""%1""==""st"" (set ANT_CMD_LINE_ARGS=%ANT_CMD_LINE_ARGS% -Dskip.junit=true) else (set ANT_CMD_LINE_ARGS=%ANT_CMD_LINE_ARGS% %1)
goto setupArgs

Now, I can just type

ant all st

Wednesday, 11 March 2009

Cannot find a valid baseurl for repo: core

I came across this error while trying to install a package on Fedora 5:

# yum install gd-devel
Loading "installonlyn" plugin
Setting up Install Process
Setting up repositories
core [1/3]
Cannot find a valid baseurl for repo: core
Error: Cannot find a valid baseurl for repo: core

The solution was relatively simple if obscure - I found it here.

I simply changed the enabled=1 to enabled=0 in:


Do you have that package installed

With yum, you can find out if you have a package installed using the query option. For example if you want to know if 'php-gd' is installed:

# rpm -q php-gd

Sunday, 22 February 2009

Mplayer sound via SPDIF

I recently lost sound via mplayer on my MythTV mediacenter. This affected any of my video collection that weren't DVD iso images since mplayer was the configured player for most types. At first I thought it might have been related to doing a system update, but now I think it was related to my tinkering which got interrupted by the kids - and then promptly forgotten about. Days or weeks later, I was surprised to find the sound missing when playing videos (rarely done - I usually watch the recordings or DVDs which use the internal player).

Now that I've fixed the issue, the problem is obvious. I have a Gigabyte motherboard and I use the SPDIF output to take the sound via optical cable into my surround sound receiver. In the video setup, I had the default player set to mplayer with the command:

mplayer -ao alsa:device=spdif -fs -zoom -quiet -vo xv %s

This directed sound to the SPDIF output. But, when using MythStream, no sound came out when watching things such as the Apple movie trailers (MythStream doesn't let you specify settings for mplayer like MythVideo does). Thus the tinkering began. What I needed to do is configure mplayer to always send sound to SPDIF. You can do this by specifying settings in ~/.mplayer/config


Now, everything is good. Mplayer sound always goes via SPDIF, whether I'm playing it through MythTV (MythVideo) or not.

Note that before I arrived at this solution I'd tried other settings in the mplayer config. When I had the settings wrong I saw the following when running mplayer from the command line:
Forced audio codec: hwac3
Cannot find codec for audio format 0x73627276.
Read DOCS/HTML/en/codecs.html!
Audio: no sound

This was occurring because I'd been googling for solutions and was lead to try values such as:

in the mplayer config file.

Wednesday, 18 February 2009

Do you need a Grails developer in Sydney?

If you haven't heard of Grails, head on over to Grails.org and have a look. Grails breaks down the barriers to the Java platform and gives you a highly productive environment for building web applications - and you are always free to utilise plain Java seamlessly.

Having used Grails on my personal projects, and written several Grails plugins, I'm sold. My time is precious and I can achieve a lot with a small amount of code, in a small amount of time - free of a lot of the bureaucracy of "Enterprise" Java applications.

If you are interested in working with Grails and need a conscientious, motivated guy on your team, drop me a line: paulrule1@gmail.com.

To get to know me better, see:

Monday, 16 February 2009

Linux sound on the Dell 1525

Sound on my Dell 1525 notebook has always been dodgy under Linux. On windows it runs fine, but under Linux (Ubuntu 8.10) sound is very low - it only becomes audible at 50% volume, so at full volume it is quiet. The real show-stopper is that Skype is unusable since the microphone is also severely attenuated. Doing a test call on Skype results in a barely audible playback.

I've spent several hours here and there, looking for solutions. The latest adventure lead me to forum posts from 2006 which lead me from resource to resource until I finally came to SoundTroubleShooting which is probably where I should have started. From here I reached the ALSA Intel supported hardware list. Turns out my hardware is not supported:

paul@dell1525:~$ lspci | grep -i audio
00:1b.0 Audio device: Intel Corporation 82801H (ICH8 Family) HD Audio Controller (rev 02)

The ICH8 is not listed as supported hardware. The solution is to get a supported sound card.

My workaround means I'm going to have to use Vista for Skype. My Dell 1525 came with Vista and I've put Ubuntu on as a dual boot option. It's been a long time since I've booted into Vista, but the sound and Skype do work as expected there. The downside to this story is that I'm going to have to prearrange Skype calls and specifically boot into Vista at these times - rather than ad-hoc calls on a whim. Kind of takes the fun out of the internet...

If I've got this wrong, please let me know, because I'd like to just do everything from Ubuntu.

Other associated information:

paul@dell1525:~$ aplay -l
**** List of PLAYBACK Hardware Devices ****
card 0: Intel [HDA Intel], device 0: STAC92xx Analog [STAC92xx Analog]
Subdevices: 1/1
Subdevice #0: subdevice #0
card 0: Intel [HDA Intel], device 3: ATI HDMI [ATI HDMI]
Subdevices: 1/1
Subdevice #0: subdevice #0

paul@dell1525:~$ uname -r

paul@dell1525:~$ cat /proc/asound/card0/codec#0 | grep -i codec
Codec: Conexant ID 2c06

The price of buying books online

I wish all books were available as PDFs. I don't like the physical space real books take up, and I don't appreciate paying for delivery, or waiting for delivery. Some books are available for Kindle, but I just want a PDF.

Recently I wanted to buy 4 books - my first instinct was to go to Amazon:

Item(s) Subtotal: AUD 97.92
Shipping & Handling: AUD 39.50
Total AUD 137.42

I'd prefer not to pay $40 for delivery, so I thought I'd check more local places to compare:

Building a WordPress Blog People Want to Read
ProBlogger: Secrets for Blogging Your Way to a Six-Figure Income
Outliers: The Story of Success
The Huffington Post Complete Guide to Blogging

So from Dymocks, 3 out of the 4 books cost AUD137.41. I don't know about delivery, I had so much trouble with the site - no response, server errors, and when trying to check out it tells me my basket is empty.

From Fishpond, all books were available, total cost of NZD182.62 with free delivery.

So Amazon wins easily. But delivery might take a month. Sigh.

Sunday, 15 February 2009

Memory upgrade for media center

I've noticed that my MythTV (Mythbuntu 8.10) machine was using 100% of its 1G and swapping. (Incidently, I was running the Blootube widescreen theme, and just switching to the MythCenter theme resulted in a snappier experience).

So, I went to buy more memory - and found that 1G cost AUD23 and a 2G stick was AUD45. Nice. I went for the 2G but when a friend commented that mixing a 1G + 2G I'd miss out on dual channel. Bummer, but I'll live with it.

When I first installed the RAM, I put the new 2G chip in the same channel (A) as the existing 1G. Booting the PC reported 3G in single channel mode, but it wouldn't boot! It simply reported a CRC error and stopped!

I took out the 1G stick, leaving just the 2G and booted okay. Hmmm, so now I put the 1G back, but in one of the channel B slots so the 2 sticks were in different channels. Now, when booting it reported 3G in Flex mode!

I'd never heard of this so I looked it up. It all gets explained very well here, and what it means is that I'll get dual channel access for the 1G stick, and the first 1G of my 2G stick, while the second 1G on the 2G stick is single channel. AWESOME! Thats just what I want! I don't have to have the same size memory sticks, but I can still benefit from dual channel when possible.

Incidentally, shortly after booting I used top to view whats going on - I saw that ALL 3G is being used and it is still swapping. Right now, it seems a bit better, with 900M free:

top - 08:26:58 up 16:53, 2 users, load average: 0.09, 0.11, 0.16
Tasks: 129 total, 1 running, 128 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3%us, 0.2%sy, 0.0%ni, 98.7%id, 0.0%wa, 0.8%hi, 0.0%si, 0.0%st
Mem: 3088208k total, 2228644k used, 859564k free, 122452k buffers
Swap: 4883720k total, 5508k used, 4878212k free, 1397300k cached

I'm not just using this machine as a media center - I'm using it as a server, running some java apps and databases etc so it's not just MythTV running here.

(I'm using a Gigabyte GA965P-S3 motherboard, see the hardware page for more information).

Wednesday, 21 January 2009

Conspiracy or oversight?

A recent Computerworld article referenced Linux, Mac OS X and Windows Vista. They linked Mac OS X and Windows Vista to their search but they didn't link Linux. Conspiracy? Oversight?

Sunday, 18 January 2009

Printing after upgrading from Mythbuntu 8.04 to 8.10

I've just noticed printing not working on my mediacenter. Its not something I do often, so I assume it stopped after the upgrade.

There was no printing configuration under settings, so I installed:

sudo apt-get install system-config-printer-gnome

This gave me UI feedback that the printer was not working.

I could see an error in /var/log/cups/error_log:
E [18/Jan/2009:00:47:46 +1100] Filter "rastertogutenprint.5.0" for printer "PSC_2100_Series" not available: No such file or directory

I found the answer by googling at http://ubuntuforums.org/archive/index.php/t-520453.html :
sudo aptitude purge gs-esp
sudo aptitude install cupsys cupsys-driver-gutenprint foomatic-db-gutenprint foomatic-filters fontconfig libtiff4 libfreetype6 gs-esp

I had to go into Settings/Printing/Server/Settings to re-enable remote printing.