I recently mapped a JPA entity to a view, and encountered this error while using hibernate to update the database schema:

ORA-01702: a view is not appropriate here

[SchemaUpdate.execute] could not complete schema update

I was using the Oracle JDBC drivers 10.1.3.3 – luckily, the solution was to upgrade the drivers – when using 11.2.0.2.0 it works fine.

http://www.oracle.com/technetwork/database/enterprise-edition/jdbc-112010-090769.html

ojdbc6.jar (2,152,051 bytes) – Classes for use with JDK 1.6. It contains the JDBC driver classes except classes for NLS support in Oracle Object and Collection types.

I’m working on a little sample framework – really only to keep my sanity and practice my chosen craft – that allows you to string together tasks in a pipeline for processing. To exercise that framework – to flush out the pros and cons of the implementation – I’m writing a sample application.

The basic idea is that each task is interested in an event. That event could be the arrival of a file, or the completion of another task etc. To string tasks together, I’ve created a simple database queue system – as a task completes, it writes to the queue and then other tasks which are interested will see that event and then begin. Tasks complete either successfully (by returning no errors) or unsuccessfully (by returning more than one error).

An important concept is to be able to add (or remove) tasks from the pipeline programmatically and without changing the database schema. So, quite simply, when a task completes, that event is written to the queue – but how do we find which events another task is interested in? And how do we do this in a way that won’t take longer over time?

So, lets assume Job2 is interested in the successful completion of Job1. We need to find all successful Job1 events which haven’t already been processed by Job2 (successfully or otherwise):

SELECT * FROM QUEUE where job='job1' and result='SUCCEEDED' and task not in (SELECT task FROM QUEUE where job='job2' and result is not null)

I’m no SQL ninja, so this is my first simple solution. It works, but it doesn’t scale. With a thousand rows, it completes pretty quickly, but with tens of thousands of rows it takes way too long (nearly 8 minutes with 40000 rows running on my Dell Inspiron laptop).

Pragmatically, we can restrict the query based on time – we don’t need to search the whole table, just the more recent events:

SELECT * FROM QUEUE where tstamp >= ? and job='job1' and result='SUCCEEDED' and task not in (SELECT task FROM QUEUE where tstamp >= ? and  job='job2' and result is not null)

You could easily just look over the events in the last hour if your tasks are short running and you get less than a thousand events an hour – it’d probably perform okay. To give an indication of performance I’ve written a little groovy script to show some basic trends – included below.

paul@paul-laptop:~$ groovy sqltest
2005 rows took: 4
4005 rows took: 5
6005 rows took: 3
8005 rows took: 3
10005 rows took: 2
12005 rows took: 3
14005 rows took: 3
16005 rows took: 4
18005 rows took: 5
20005 rows took: 3


Notice though, that this is a great example of where indexes really help – without the index times get larger as the rowcount gets larger:

paul@paul-laptop:~$ groovy sqltest
2005 rows took: 13
4005 rows took: 34
6005 rows took: 123
8005 rows took: 134
10005 rows took: 229
12005 rows took: 237
14005 rows took: 330
16005 rows took: 342
18005 rows took: 428
20005 rows took: 442


Its a pity I have to worry about the time restriction, but it does make the solution workable and at least in my intended application, appropriate. I was hoping to use the SQL MINUS function, but it appears MYSQL doesn’t support it.

Here’s the groovy script I used to generate the results. The difference between using the time restriction and the index is so dramatic and obvious, it is a great example.

import groovy.sql.Sql
import groovy.grape.Grape

Grape.grab(group:'mysql', module:'mysql-connector-java', version:'5.1.10', classLoader: this.class.classLoader.rootLoader)

enum STATUS { SUCCEEDED, FAILED }

def go() {
	def sql = Sql.newInstance("jdbc:mysql://localhost:3306/spike", "spike","password", "com.mysql.jdbc.Driver")
	try {
	   sql.execute("drop table QUEUE")
	} catch(Exception e){}

	sql.execute("CREATE TABLE QUEUE (id INTEGER  NOT NULL,job varchar (20) NOT NULL,task varchar (20) NOT NULL,result VARCHAR (20) NOT NULL, tstamp timestamp, PRIMARY KEY (id)) ENGINE = MyISAM")
	sql.execute("create index idx1 on QUEUE(tstamp, job, result)")

	(1..10).each() {
		def d = createHistory(sql,1000*it)
		findJobsToProcess(sql, d)
	}
}

def findJobsToProcess(sql, d) {
	def rowcount = 0
	sql.eachRow("select count(*) from QUEUE") { row ->
		rowcount = row[0]
	}

	long start, end

	start = System.currentTimeMillis()
	// now find the 'job1' items that haven't been processed by 'job2'
	sql.eachRow("SELECT * FROM QUEUE where tstamp >= ? and job='job1' and result=? and task not in (SELECT task FROM QUEUE where tstamp >= ? and  job='job2' and result is not null) order by tstamp asc", [d,STATUS.SUCCEEDED.name(),d]) {
//	    println "${it.id} ${it.job} ${it.task} ${it.result} ${it.tstamp}"
	}
	end = System.currentTimeMillis()
	println "${rowcount} rows took: ${end-start}"

}

def createHistory(sql, count) {
	sql.execute("truncate table QUEUE")
	int id = 1
	int task = 1
	// create a "history" of previous jobs
	STATUS.each() { status ->
		(1..count/2).each() {
			sql.execute("insert into QUEUE (id,job,task,result,tstamp) values (?,?,?,?,?)", [id++,'job1',task.toString(),status.name(), new Date()])
			sql.execute("insert into QUEUE (id,job,task,result,tstamp) values (?,?,?,?,?)", [id++,'job2',task.toString(),status.name(), new Date()])
			task++
		}
	}
	sleep(1000)
	def d = new Date()
	// insert some successful 'job1' items which 'job2' hasn't processed
	(1..5).each() {
		sql.execute("insert into QUEUE (id,job,task,result,tstamp) values (?,?,?,?,?)", [id++,'job1',task.toString(),STATUS.SUCCEEDED.name(), new Date()])
		task++
	}
	return  d
}

go()

In running this test, I used:

• Dell Inspiron 1525, Intel(R) Core(TM)2 Duo Processor T5550, 1.83 GHz, 2MB Cache, 667 MHz FSB, 2GB (2 X 1024MB) 667MHz Dual Channel DDR2 SDRAM, 160GB 7200RPM Performance Hard Drive
• Ubuntu 9.10 32 bit desktop edition
• Groovy Version: 1.6.5
• JVM: 1.6.0_03
• mysql Ver 14.14 Distrib 5.1.37, for debian-linux-gnu (i486)

Sometimes applications need access to data from another application. Here is one approach you could take:

1. Define a view in the database that owns the data – this view should expose the data that the external application needs
2. Grant select permissions so that the external application can select from the view
3. Create a synonym in the external applications database for the view so that it appears as a local resource

This has the advantage that the owner of the data defines what is exposed. Handy if the data is sensitive in any way.

This solution also assumes that the external application only wants to read the data. If it wanted to write to the database, then the application that owns the data should probably publish an API (eg web services et al).

It can be quite frustrating when you get exceptions like those below – constraint violations where the constraint has a system generated name which means nothing to you:

2009-03-23 23:30:43 ERROR [AbstractFlushingEventListener.performExecutions] Could not synchronize database state with session
org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update
…
Caused by: java.sql.BatchUpdateException: ORA-00001: unique constraint (SAMPLE.SYS_C00123456) violated
…

The easiest way to find out which table this constraint is associated with is to issue a statement as shown:

select constraint_name, table_name from user_constraints where constraint_name='SYS_C00123456';

Now you know the table involved, you can easily find out the exact nature of the constraint in various ways – my tool of choice is SqlDeveloper – a GUI interface to Oracle (and other) database.

I’ve been wanting a way to use playlists on my media center either from within MythTV or with any other player. The reason is I have a bunch of short videos (music and documentaries) that I’d like to be able to play consecutively.

It turns out to be a case of just RTFM. You can create playlists for MPlayer from directory listings, and then associate playlists with a particular mplayer command.

To generate the playlist from all files in a directory, I use the following:

dir * | sed 's/\\//g' > all_videos.pls

The sed part of this command removes the \ from escaped spaces… i.e. a file ‘Hello world’ is output from the dir command as ‘Hello\ world’ – the space is escaped. We need to generate a file without these escaped spaces for it to work properly.

I use the following mplayer command associated with the pls extension:

mplayer -shuffle -fs -zoom -quiet -vo xv -playlist %s

I was just using a query which made use of the ST_INTERSECTS function:

select * from table1 where st_intersects(st_point( ?, ?, 1),shape)=1

With the data I had, this query took 30 seconds! Before launching into an investigation to find out why, I just decided to swap the parameters – this made all the difference:

select * from table1 where st_intersects(shape, st_point( ?, ?, 1))=1

Now the query returns instantly! I’m no database expert, so investigating why the first version of the query took so long would have been a waste of valuable time – when such a simple solution was at hand.

Often there is a need to compare two databases and see the differences. I come across this a lot when releasing a new build into an existing environment – the new code runs on the development database, but the test environment needs a schema upgrade before the code will run.

There are tools that will compare schemas for you, but sometimes I just need to quickly find out what columns are missing from the target schema (this is particularly useful when comparing different development schemas and you want to quickly implement changes – production releases need much better quality control than this).

The sql below provides one solution – showing the columns that are missing from ‘owner1′ when compared to ‘owner2′ for tables that match the prefix:

 SELECT table_name, column_name, data_type FROM all_tab_cols
  WHERE lower(owner)=lower(:owner1)
AND table_name LIKE :prefix
AND column_name NOT LIKE 'SYS%'
AND table_name||'-'||column_name NOT IN
  (SELECT table_name||'-'||column_name FROM all_tab_cols
    WHERE lower(owner)=lower(:owner2)
  )
ORDER BY table_name, column_name;

This will ofcourse identify missing tables. It’s primative, but its an easy way to see basic structural differences using SQL – without the need for expensive tools. Since it is basic sql, you could include this as part of an environment verification test suite, or include in your application startup procedure – providing you have access to a reference schema for comparison.

I use SQLDeveloper when interacting with Oracle. When viewing tables with date columns it can be frustrating that the default display does not show time (just day, month and year).

To change this behaviour, you can set the date format for your current session:

alter session set NLS_DATE_FORMAT='DD-Mon-YYYY HH24:MI:SS'

Now, when you view tables or resultsets, you’ll have a more precise view of the data:

04-Sep-2008 09:07:51

It frequently surprises me when seemingly simple things are missing from mature products. For example, Oracle doesn’t have a boolean type. Strange but apparently true.

Never mind, this article describes how to work around this.

When making support requests it is always helpful to include version information about the product in question. Rather than just stating ‘version X’ I like to get the software to display the version information and just copy and paste it – this way there can be no confusion, and there may even be extra useful information.

So, how to display the oracle version information? I came across this useful post which suggests:

select banner from v$version;

which on the instance I’m looking at produces:

BANNER----------------------------------------------------------------
Oracle10g Enterprise Edition Release 10.2.0.3.0 - Production
PL/SQL Release 10.2.0.3.0 - Production
CORE    10.2.0.3.0       Production
TNS for Solaris: Version 10.2.0.3.0 - Production
NLSRTL Version 10.2.0.3.0 - Production

Being an SQL query is even more useful than a command line, since you can run this within your applications if you need to display such data.