I recently mapped a JPA entity to a view, and encountered this error while using hibernate to update the database schema:

ORA-01702: a view is not appropriate here

[SchemaUpdate.execute] could not complete schema update

I was using the Oracle JDBC drivers 10.1.3.3 – luckily, the solution was to upgrade the drivers – when using 11.2.0.2.0 it works fine.

http://www.oracle.com/technetwork/database/enterprise-edition/jdbc-112010-090769.html

ojdbc6.jar (2,152,051 bytes) – Classes for use with JDK 1.6. It contains the JDBC driver classes except classes for NLS support in Oracle Object and Collection types.

I went to a Agile Sydney meetup recently – “Automated web tests as a team communication tool” – by John Smart of Wakaleo Consulting. It was great stuff, and the presentation is now online.

There is a nice story here in that the project feature set is documented via
easily readable and executable tests, and the reporting shows which features are
implemented and which aren’t.

More information on the the Thucydides tool he references can be found at:

• https://github.com/thucydides-webtests/thucydides/wiki

While reading that presentation online, I found more gold here in some of his other presentations:

• JUnit Kung Fu: Getting More Out of Your Unit Tests
• Introduction to Domain Driven Design

They are a great read and are great pointers in the right direction to improve your software development.

(If you haven’t seen them before, check out Wakaleo’s training courses. I haven’t been on them, but based on the course content, they look to be the most relevant courses I’ve come across)

The last couple of projects I’ve been on I’ve been frustrated by a few things about the code base that I couldn’t quite put my finger on, and the DDD presentation hit a nerve.

I searched around for more about DDD and found some more excellent resources:

• Domain driven design quickly
• DDD sample application
• Time and Money

I still have to absorb all of this material and try it out but I can instantly recognise that it addresses some if not most of the code problems that have frustrated me.

So, exciting times are ahead. Its always refreshing to find new tricks and efficient ways of improving software development!

I recently had to investigate how to generate a PDF from a Crystal Report created by another team. Without knowing anything about Crystal Reports, I had to google around for information and piece it all together. It turns out to be really simple once you know how.

We were using Business Objects 4.0, and probably the most important thing was to get the Java library – download ‘SAP Crystal Reports for Java runtime components – Java Reporting Component (JRC)’ from
http://www.businessobjects.com/campaigns/forms/downloads/crystal/eclipse/datasave.asp

There are a lot of samples on the web to look at – you might find something to help here:
http://wiki.sdn.sap.com/wiki/display/BOBJ/Java+Reporting+Component++SDK+Samples

A good example to start with is “JRC EXPORT REPORT”:
http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/40d580ce-bd66-2b10-95b0-cc4d3f2dcaef

If you have an RPT file, the java to generate the report is relatively simple – :

ReportClientDocument reportClientDoc = new ReportClientDocument();
reportClientDoc.open("My crystal report.rpt", 0);
ByteArrayInputStream byteArrayInputStream = (ByteArrayInputStream)reportClientDoc.getPrintOutputController().export(ReportExportFormat.PDF);
reportClientDoc.close();

The report I dealt with was developed by another team, and expected a data source called ‘TESTDB’ to be available. Without that data source present, I would get the following error:

com.crystaldecisions.sdk.occa.report.lib.ReportSDKException: Error finding JNDI name (TESTDB)---- Error code:-2147467259 Error code name:failed

Setting a data source up in Tomcat is trivial, just by adding a resource to context.xml:

<Context>
    <WatchedResource>WEB-INF/web.xml</WatchedResource>

    <Resource name="jdbc/TESTDB" auth="Container" type="javax.sql.DataSource"
               maxActive="100" maxIdle="30" maxWait="10000"
               username="user" password="passwd" driverClassName="oracle.jdbc.OracleDriver"
               url="jdbc:oracle:thin:@dbserver:1521:db1"/>
</Context>

Now the data source is set up, there’s another problem – the report expects a parameter and we get this error:

com.crystaldecisions.sdk.occa.report.lib.ReportSDKParameterFieldException: InternalFormatterException---- Error code:-2147217394 Error code name:missingParameterValueError

Adding the report parameter is simple once you know how:

ParameterFieldController paramController = reportClientDoc.getDataDefController().getParameterFieldController();
paramController.setCurrentValue("","MyParamName","MyParamValue");

There’s probably a lot of subtle information and detail missing here, but it works – I can generate the PDF via java code so its now at a stage where it can be integrated into our application.

So, the final spike test code looks like:

<%@page contentType="text/html"%>
<%@page pageEncoding="UTF-8"%>

<%//Crystal Java Reporting Component (JRC) imports.%>
<%@page import="com.crystaldecisions.reports.sdk.*" %>
<%@page import="com.crystaldecisions.sdk.occa.report.lib.*" %>
<%@page import="com.crystaldecisions.sdk.occa.report.exportoptions.*" %>

<%//Java imports. %>
<%@page import="java.io.*" %>

<%

try {

	//Open report.
	ReportClientDocument reportClientDoc = new ReportClientDocument();
	reportClientDoc.open("MyReport.rpt", 0);
	ParameterFieldController paramController = reportClientDoc.getDataDefController().getParameterFieldController();
	paramController.setCurrentValue("","MyParamName","MyParamValue");
	ByteArrayInputStream byteArrayInputStream = (ByteArrayInputStream)reportClientDoc.getPrintOutputController().export(ReportExportFormat.PDF);
	reportClientDoc.close();

	writeToBrowser(byteArrayInputStream, response, "application/pdf");

} catch(Exception ex) {
	out.println(ex);
}
%>

<%!
   /*
	* Utility method that demonstrates how to write an input stream to the server's local file system.
	*/
	private void writeToBrowser(ByteArrayInputStream byteArrayInputStream, HttpServletResponse response, String mimetype) throws Exception {

		//Create a byte[] the same size as the exported ByteArrayInputStream.
		byte[] buffer = new byte[byteArrayInputStream.available()];
		int bytesRead = 0;

		//Set response headers to indicate mime type and inline file.
		response.reset();
		response.setHeader("Content-disposition", "inline;filename=report.pdf");
		response.setContentType(mimetype);

		//Stream the byte array to the client.
		while((bytesRead = byteArrayInputStream.read(buffer)) != -1) {
			response.getOutputStream().write(buffer, 0, bytesRead);
		}

		//Flush and close the output stream.
		response.getOutputStream().flush();
		response.getOutputStream().close();

	}
%>

Note, implementing this in a JSP was just a simple and quick shortcut to investigate the API.

I put a simple CRConfig.xml in WEB-INF/classes:

<?xml version="1.0" encoding="utf-8"?>
<CrystalReportEngine-configuration>
    <reportlocation>..</reportlocation>
    <timeout>0</timeout>
    <ExternalFunctionLibraryClassNames>
    	<classname></classname>
    </ExternalFunctionLibraryClassNames>
</CrystalReportEngine-configuration>

Note that this specified a report location of .. which meant I had to put MyReport.rpt in the WEB-INF directory of my application (i.e. [web-root]/WEB-INF/classes/.. = [web-root]/WEB-INF) – contents of the WEB-INF directory should not be available for download.

If the rpt file cannot be found, you’ll see an error like:

com.crystaldecisions.sdk.occa.report.lib.ReportSDKException: Report file /[path-to-webapps]/webapps/cr/WEB-INF/MyReport.rpt not found---- Error code:-2147215356 Error code name:fileNotOpened

Back in the day it used to be a bit more difficult than it is now to dynamically generate your java application classpath – you’d have to write a script to loop over all the jar files in a directory, appending them to your classpath variable – like this.

Since Java 6 though, its been a lot easier – you can use wildcards to specify all jars in a directory. See the “Understanding class path wildcards” in the Java SE 6 documentation.

Now its as simple as:

java -cp /my/lib/dir/* MyClass

Simple! As it should be!

I’ve been developing a simple little command line application and one of the libraries I’m using seems to log debug information to the console. To clean up the console output, I had to turn off the JDK by using:

        LogManager.getLogManager().reset();

This seemed to do the trick for me, but it sounds like in some cases you may need to go a bit further and turn off logging at the global logger level:

        LogManager.getLogManager().reset();
        Logger globalLogger = Logger.getLogger(java.util.logging.Logger.GLOBAL_LOGGER_NAME);
        globalLogger.setLevel(java.util.logging.Level.OFF);

I’ve been writing JUnit Parameterized tests (a nice intro here – also see Theories) to do some data driven API testing. Now, when introducing Spring into the mix there are a couple of extra things to do. I came unstuck though because I was trying to do Transactional tests – since I’m now using @RunWith(Parameterized.class) and setting up my Spring TestContextManager manually the @Transaction annotations caused an exception:

java.lang.IllegalArgumentException: Superclass has no null constructors but no arguments were given

I couldn’t find any built in solution, so I’ve gone with manual transaction management in my test, using doInTransaction:

@RunWith(Parameterized.class)
@ContextConfiguration(locations = "classpath*:/testContext.xml")
public class MyTest {

    @Autowired
    PlatformTransactionManager transactionManager;

    private TestContextManager testContextManager;

    public MyTest (... parameters for test) {
        // store parameters in instance variables
    }

    @Before
    public void setUpSpringContext() throws Exception {
        testContextManager = new TestContextManager(getClass());
        testContextManager.prepareTestInstance(this);
    }

    @Parameterized.Parameters
    public static Collection generateData() throws Exception {
        ArrayList list = new ArrayList();
        // add data for each test here
        return list;
    }

    @Test
    public void validDataShouldLoadFully() throws Exception {
        new TransactionTemplate(transactionManager).execute(new TransactionCallback() {
            public Object doInTransaction(TransactionStatus status) {
                status.setRollbackOnly();
                try {
                    ... do cool stuff here

                } catch (Exception e) {
                    throw new RuntimeException(e);
                }
                return null;
            }
        });

    }

I spend a lot of time on the train travelling to work. Working from home has not really taken off in this part of the world – so, what better to do with that time than practice my craft? Its not comfortable, its hard to concentrate, but progress can be made. I have successfully used this tool to observe real performance problems, and confirm they’ve been resolved.

JtSysMon is a very simple monitoring tool which will poll resources and time the response. The response time is logged to a database, and from there, you can see which resources are either slow or failing/erroring.

I wrote this because I needed a way to make sure that various resources were available and responsive during development, so that we could respond quickly (proactively) to problems and have them fixed before developers or testers noticed or wasted time fault finding.

Existing solutions (Nagios, Zenoss etc) seemed too complex for such a simple requirement, with too much of a learning curve and setup required for essentially what is polling HTTP and SQL (please correct me if I am wrong).

Install

Download from http://www.javathinking.com/downloads/. Now extract the binary archive so that you have a directory structure like:

- jtsysmon
|- bin
|- conf
|- lib

By default, an embedded instance of the DERBY database will be used to store the configuration and poll results. If you want, you can edit conf/database.properties to point to another database. If you do, you will have to add the database driver jar file to the lib dir.

Configure

JtSysMon currently comes with 2 monitor types – SqlMonitor and HttpMonitor.

To configure a monitor, specify the class, a name, and then properties of the class as required. Reflection is used to set the properties of the class:

sh run.sh add name=sql1 class=com.javathinking.jtsysmon.core.monitor.SqlMonitor driverClass=org.hibernate.dialect.DerbyDialect connectionString=jdbc:derby://localhost:1527/jtsysmon “sql=select count(*) from APP.MONITORCONFIG”

sh run.sh add name=http1 class=com.javathinking.jtsysmon.core.monitor.HttpMonitor url=http://localhost/ checkFor=Hello

To list the monitors, use:

sh run.sh -list

Listing monitors

1048576 sql1
class = class com.javathinking.jtsysmon.core.monitor.SqlMonitor
connectionString = jdbc:derby://localhost:1527/jtsysmon
driverClass = org.hibernate.dialect.DerbyDialect
sql = select count(*) from APP.MONITORCONFIG

To delete a monitor, use delete, with the ID of the monitor (shown in the listing above):

sh run.sh -delete 1048576

To start monitoring, use:

sh run.sh -start

Use CTRL-C to stop.

To view alarms (where the time taken to perform poll exceeds a specified value) or problems (where the poll failed or errored) use:

sh run.sh -alarms -threshold 5 -period 20

Listing events over last 20 minutes

ALARMS (duration > 5)

19:08:18 paul-laptop1 sql1 28ms SUCCESS

PROBLEMS (non-successful events)

19:08:21 paul-laptop1 http1 90ms  ERROR

Summary

So there you have it – a bare bones way to check connectivity, response times, and availability of resources in your environment. This is meant only as a development tool, NOT for monitoring production resources.

Futures

If there is any interest from the community in using this tool, you could easily imagine:

• multiple agents logging to a central database
• more monitors
  • snmp
  • ssh
  • scripts
• a web client
  • to display alarms and graphs of response times
  • to configure the monitors
• notifiers to alert users of problems

For the latest information, see the source code (and README.TXT) at:

• http://javathinking.googlecode.com/svn/trunk/javathinking-sysmon/

I’m working on a little sample framework – really only to keep my sanity and practice my chosen craft – that allows you to string together tasks in a pipeline for processing. To exercise that framework – to flush out the pros and cons of the implementation – I’m writing a sample application.

The basic idea is that each task is interested in an event. That event could be the arrival of a file, or the completion of another task etc. To string tasks together, I’ve created a simple database queue system – as a task completes, it writes to the queue and then other tasks which are interested will see that event and then begin. Tasks complete either successfully (by returning no errors) or unsuccessfully (by returning more than one error).

An important concept is to be able to add (or remove) tasks from the pipeline programmatically and without changing the database schema. So, quite simply, when a task completes, that event is written to the queue – but how do we find which events another task is interested in? And how do we do this in a way that won’t take longer over time?

So, lets assume Job2 is interested in the successful completion of Job1. We need to find all successful Job1 events which haven’t already been processed by Job2 (successfully or otherwise):

SELECT * FROM QUEUE where job='job1' and result='SUCCEEDED' and task not in (SELECT task FROM QUEUE where job='job2' and result is not null)

I’m no SQL ninja, so this is my first simple solution. It works, but it doesn’t scale. With a thousand rows, it completes pretty quickly, but with tens of thousands of rows it takes way too long (nearly 8 minutes with 40000 rows running on my Dell Inspiron laptop).

Pragmatically, we can restrict the query based on time – we don’t need to search the whole table, just the more recent events:

SELECT * FROM QUEUE where tstamp >= ? and job='job1' and result='SUCCEEDED' and task not in (SELECT task FROM QUEUE where tstamp >= ? and  job='job2' and result is not null)

You could easily just look over the events in the last hour if your tasks are short running and you get less than a thousand events an hour – it’d probably perform okay. To give an indication of performance I’ve written a little groovy script to show some basic trends – included below.

paul@paul-laptop:~$ groovy sqltest
2005 rows took: 4
4005 rows took: 5
6005 rows took: 3
8005 rows took: 3
10005 rows took: 2
12005 rows took: 3
14005 rows took: 3
16005 rows took: 4
18005 rows took: 5
20005 rows took: 3


Notice though, that this is a great example of where indexes really help – without the index times get larger as the rowcount gets larger:

paul@paul-laptop:~$ groovy sqltest
2005 rows took: 13
4005 rows took: 34
6005 rows took: 123
8005 rows took: 134
10005 rows took: 229
12005 rows took: 237
14005 rows took: 330
16005 rows took: 342
18005 rows took: 428
20005 rows took: 442


Its a pity I have to worry about the time restriction, but it does make the solution workable and at least in my intended application, appropriate. I was hoping to use the SQL MINUS function, but it appears MYSQL doesn’t support it.

Here’s the groovy script I used to generate the results. The difference between using the time restriction and the index is so dramatic and obvious, it is a great example.

import groovy.sql.Sql
import groovy.grape.Grape

Grape.grab(group:'mysql', module:'mysql-connector-java', version:'5.1.10', classLoader: this.class.classLoader.rootLoader)

enum STATUS { SUCCEEDED, FAILED }

def go() {
	def sql = Sql.newInstance("jdbc:mysql://localhost:3306/spike", "spike","password", "com.mysql.jdbc.Driver")
	try {
	   sql.execute("drop table QUEUE")
	} catch(Exception e){}

	sql.execute("CREATE TABLE QUEUE (id INTEGER  NOT NULL,job varchar (20) NOT NULL,task varchar (20) NOT NULL,result VARCHAR (20) NOT NULL, tstamp timestamp, PRIMARY KEY (id)) ENGINE = MyISAM")
	sql.execute("create index idx1 on QUEUE(tstamp, job, result)")

	(1..10).each() {
		def d = createHistory(sql,1000*it)
		findJobsToProcess(sql, d)
	}
}

def findJobsToProcess(sql, d) {
	def rowcount = 0
	sql.eachRow("select count(*) from QUEUE") { row ->
		rowcount = row[0]
	}

	long start, end

	start = System.currentTimeMillis()
	// now find the 'job1' items that haven't been processed by 'job2'
	sql.eachRow("SELECT * FROM QUEUE where tstamp >= ? and job='job1' and result=? and task not in (SELECT task FROM QUEUE where tstamp >= ? and  job='job2' and result is not null) order by tstamp asc", [d,STATUS.SUCCEEDED.name(),d]) {
//	    println "${it.id} ${it.job} ${it.task} ${it.result} ${it.tstamp}"
	}
	end = System.currentTimeMillis()
	println "${rowcount} rows took: ${end-start}"

}

def createHistory(sql, count) {
	sql.execute("truncate table QUEUE")
	int id = 1
	int task = 1
	// create a "history" of previous jobs
	STATUS.each() { status ->
		(1..count/2).each() {
			sql.execute("insert into QUEUE (id,job,task,result,tstamp) values (?,?,?,?,?)", [id++,'job1',task.toString(),status.name(), new Date()])
			sql.execute("insert into QUEUE (id,job,task,result,tstamp) values (?,?,?,?,?)", [id++,'job2',task.toString(),status.name(), new Date()])
			task++
		}
	}
	sleep(1000)
	def d = new Date()
	// insert some successful 'job1' items which 'job2' hasn't processed
	(1..5).each() {
		sql.execute("insert into QUEUE (id,job,task,result,tstamp) values (?,?,?,?,?)", [id++,'job1',task.toString(),STATUS.SUCCEEDED.name(), new Date()])
		task++
	}
	return  d
}

go()

In running this test, I used:

• Dell Inspiron 1525, Intel(R) Core(TM)2 Duo Processor T5550, 1.83 GHz, 2MB Cache, 667 MHz FSB, 2GB (2 X 1024MB) 667MHz Dual Channel DDR2 SDRAM, 160GB 7200RPM Performance Hard Drive
• Ubuntu 9.10 32 bit desktop edition
• Groovy Version: 1.6.5
• JVM: 1.6.0_03
• mysql Ver 14.14 Distrib 5.1.37, for debian-linux-gnu (i486)

I just recently noticed that Amazon have changed the way you invoke their Product Advertising API – you now need to sign requests as described in the Example REST Requests.

I obviously haven’t been reading my emails, since the first clue came when I saw:

<Error><Code>MissingParameter</Code><Message>The request must contain the parameter Signature.</Message></Error>

Luckily, they have a java example demonstrating how to make an API request.

If you extract the source from this sample into your <GRAILS_APP>/src/java directory, then you can invoke the amazon api with code like:

import org.codehaus.groovy.grails.commons.ConfigurationHolder
import com.amazon.advertising.api.sample.*

class AmazonService {
  private static final String ENDPOINT = "ecs.amazonaws.com";
  private static final String AWS_SECRET_KEY = "put your secret key here";
  private static final String AWS_ACCESS_KEY_ID = "put your access key here";

  boolean transactional = false

  /**
   * Call amazon lookup api.
   * Returns xml document.
   */
  def getData(String asin) {
    SignedRequestsHelper helper = SignedRequestsHelper.getInstance(ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_KEY);

    String requestUrl = null;
    String title = null;

    Map params = new HashMap();
    params.put("Service", "AWSECommerceService");
    params.put("Version", "2009-03-31");
    params.put("Operation", "ItemLookup");
    params.put("ItemId", asin);
    params.put("ResponseGroup", "Small,Medium");

    requestUrl = helper.sign(params);

    def xml = new URL(requestUrl).text
    return new XmlSlurper().parseText(xml)
  }
}

You could, of-course, define your access keys in the Config.groovy class if you prefer.

(To find your identifiers, see http://docs.amazonwebservices.com/AWSECommerceService/latest/DG/index.html?ViewingCredentials.html)

Recently I was working with some code which defined beans as request scope. Junit tests failed with the following exception while trying to initialize the application context:

Caused by: org.springframework.beans.factory.BeanCreationException: Could not autowire field: …; nested exception is java.lang.IllegalStateException: No Scope registered for scope ‘request’

I found the solution by searching around (unfortunately I didn’t record the sources). Basically, you can register the required scopes as shown below:

import org.springframework.mock.web.MockHttpServletRequest;
import org.springframework.web.context.request.RequestContextHolder;
import org.springframework.web.context.request.RequestScope;
import org.springframework.web.context.request.ServletRequestAttributes;
import org.springframework.web.context.request.SessionScope;

public class SampleTest extends AbstractDependencyInjectionSpringContextTests {

    @Override
    protected void prepareTestInstance() throws Exception {
        context.getBeanFactory().registerScope("session", new SessionScope());
        context.getBeanFactory().registerScope("request", new RequestScope());
        MockHttpServletRequest request = new MockHttpServletRequest();
        attributes = new ServletRequestAttributes(request);
        RequestContextHolder.setRequestAttributes(attributes);

        super.prepareTestInstance();
    }
}