Postgres Transaction Deadlock

After some changes recently, my application began to hang. I run the Glassfish V2 application server and so went for a stacktrace to ascertain the responsible Java code:

asadmin generate-jvm-report --type=thread

This showed that the server was waiting on the database to return from an update query. This rang alarm bells - a locking wait I suspect...

I decided to check with Postgres which query was waiting:

SELECT procpid, current_query FROM pg_stat_activity;

This showed me that one particular update to my c_account table was waiting.

I was using two transactions: 1 & 2. The code which started tx1 then starts tx2 before committing tx1.

Transaction 1:
Inserts a logging record, referencing c_account with a foreign key.

Transaction 2:
Changes the account bar flag on c_account with an SQL update.

This scenario caused a deadlock, since tx1 takes a row-exclusive lock on the log table and any rows referenced by its foreign keys. When I go to modify the flag on c_account, then tx2 has to wait until tx1 has completed and of course, this will never happen.

Two methods to get around this:
1) Better transaction demarcation. Carefully avoid insert/update referencing the same entity.
2) More locking-aware table design. Have a one-one table for flags such that new records may reference the core entity, without locking the one-one tuple.

For now I am opting for option 1, but will keep option 2 in mind for future table design.

Spring -Junit - Testing

Create unit tests the Spring way.

This requires version 4.4 of junit (for v2.5.6 of Spring)
Add the SpringTest module to the test libs

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = "classpath:/MyTestConfig.xml")
public class MyClassTest extends AbstractTransactionalJUnit4SpringContextTests // For Tx enabled tests

Can annotate methods with @NotTransactional to not have a TX created for a method in the test. Methods called inside will still
create TX if they require one.

Don't forget to create your own EntityManager of SessionManager for your persistance.


<context:annotation-config />
<context:component-scan base-package="com.stuff"/>
<aop:aspectj-autoproxy/>

<tx:annotation-driven transaction-manager="transactionManager" />

<bean class="org.springframework.orm.jpa.vendor.HibernateJpaDialect"/>
<bean class="org.springframework.orm.jpa.support.PersistenceAnnotationBeanPostProcessor" />
<bean class="org.springframework.dao.annotation.PersistenceExceptionTranslationPostProcessor"/>


<bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="entityManagerFactory" ref="entityManagerFactory" />
</bean>

<bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="persistenceXmlLocation" value="classpath:persistence-test.xml"/>
<property name="dataSource" ref="dataSource" />
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
<property name="database" value="SQL_SERVER" />
<property name="showSql" value="true" />
</bean>
</property>
<property name="jpaProperties">
<props>
<prop key="hibernate.dialect">org.hibernate.dialect.SQLServerDialect</prop>
<prop key="hibernate.hbm2ddl.auto">update</prop>
</props>
</property>
</bean>

<bean id="dataSource" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="com.microsoft.sqlserver.jdbc.SQLServerDriver" />
<property name="url" value="jdbc:sqlserver://127.0.0.1:1433;databaseName=My-TEST" />
<property name="username" value="UserName" />
<property name="password" value="Password" />
</bean>

Strange classloader bomb-out in Glassfish V2

LDR5207: EJBClassLoader EJBClassLoader :
doneCalled = true
doneSnapshot = EJBClassLoader.done() called ON EJBClassLoader :
urlSet = [URLEntry : file:/Users/neilbeveridge/Library/glassfish/domains/domain1/lib/classes/]
doneCalled = false
Parent -> ASChain parentCL :: Addons Chain parentCL :: Shared ClassLoader Chain parentCL :: sun.misc.Launcher$AppClassLoader@6d6f0472 URLs ::
, file:/Users/neilbeveridge/Library/glassfish/lib/javaee.jar, file:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/tools.jar, file:/Users/neilbeveridge/Library/glassfish/lib/install/applications/jmsra/imqjmsra.jar, file:/Users/neilbeveridge/Library/glassfish/lib/com-sun-commons-launcher.jar, file:/Users/neilbeveridge/Library/glassfish/lib/com-sun-commons-logging.jar, file:/Users/neilbeveridge/Library/glassfish/imq/lib/jaxm-api.jar, file:/Users/neilbeveridge/Library/glassfish/imq/lib/fscontext.jar, file:/Users/neilbeveridge/Library/glassfish/imq/lib/imqbroker.jar, file:/Users/neilbeveridge/Library/glassfish/imq/lib/imqjmx.jar, file:/Users/neilbeveridge/Library/glassfish/imq/lib/imqxm.jar, file:/Users/neilbeveridge/Library/glassfish/lib/webservices-rt.jar, file:/Users/neilbeveridge/Library/glassfish/lib/webservices-tools.jar, file:/Users/neilbeveridge/Library/glassfish/lib/mail.jar, file:/Users/neilbeveridge/Library/glassfish/lib/appserv-jstl.jar, file:/Users/neilbeveridge/Library/glassfish/lib/jmxremote_optional.jar, file:/Users/neilbeveridge/Library/glassfish/lib/SUNWjdmk/5.1/lib/jdmkrt.jar, file:/Users/neilbeveridge/Library/glassfish/lib/activation.jar, file:/Users/neilbeveridge/Library/glassfish/lib/appserv-rt.jar, file:/Users/neilbeveridge/Library/glassfish/lib/appserv-admin.jar, file:/Users/neilbeveridge/Library/glassfish/lib/appserv-cmp.jar, file:/Users/neilbeveridge/Library/glassfish/updatecenter/lib/updatecenter.jar, file:/Users/neilbeveridge/Library/glassfish/jbi/lib/jbi.jar, file:/Users/neilbeveridge/Library/glassfish/imq/lib/imqjmx.jar, file:/Users/neilbeveridge/Library/glassfish/lib/ant/lib/ant.jar, file:/Users/neilbeveridge/Library/glassfish/lib/dbschema.jar, file:/Users/neilbeveridge/Library/glassfish/lib/appserv-deployment-client.jar, file:/Users/neilbeveridge/Library/glassfish/lib/appserv-ee.jar, file:/Users/neilbeveridge/Library/glassfish/lib/appserv-ext.jar, file:/Users/neilbeveridge/Library/glassfish/lib/appserv-jwsacc.jar, file:/Users/neilbeveridge/Library/glassfish/lib/appserv-launch.jar, file:/Users/neilbeveridge/Library/glassfish/lib/appserv-se.jar, file:/Users/neilbeveridge/Library/glassfish/lib/appserv-tags.jar, file:/Users/neilbeveridge/Library/glassfish/lib/appserv-ws.jar, file:/Users/neilbeveridge/Library/glassfish/lib/asm-3.1.jar, file:/Users/neilbeveridge/Library/glassfish/lib/j2ee.jar, file:/Users/neilbeveridge/Library/glassfish/lib/jackson-asl-0.9.4.jar, file:/Users/neilbeveridge/Library/glassfish/lib/jersey-bundle-1.0.3.1.jar, file:/Users/neilbeveridge/Library/glassfish/lib/jesmf-plugin.jar, file:/Users/neilbeveridge/Library/glassfish/lib/jettison-1.0.1.jar, file:/Users/neilbeveridge/Library/glassfish/lib/jhall.jar, file:/Users/neilbeveridge/Library/glassfish/lib/jmac-api.jar, file:/Users/neilbeveridge/Library/glassfish/lib/jsf-impl.jar, file:/Users/neilbeveridge/Library/glassfish/lib/jsr311-api-1.0.jar, file:/Users/neilbeveridge/Library/glassfish/lib/jxta.jar, file:/Users/neilbeveridge/Library/glassfish/lib/Pack200Task.jar, file:/Users/neilbeveridge/Library/glassfish/lib/shoal-gms.jar, file:/Users/neilbeveridge/Library/glassfish/lib/toplink-essentials-agent.jar, file:/Users/neilbeveridge/Library/glassfish/lib/toplink-essentials.jar, file:/Users/neilbeveridge/Library/glassfish/
constituent CLs ::
constituent CLs ::
:: asimpl :: optionalChain
AT Mon Jul 19 16:08:56 BST 2010
BY :com.sun.enterprise.loader.EJBClassLoader.printStackTraceToString(EJBClassLoader.java:813)
com.sun.enterprise.loader.EJBClassLoader.done(EJBClassLoader.java:173)
com.sun.enterprise.deployment.backend.Deployer.releaseClassLoader(Deployer.java:585)
com.sun.enterprise.deployment.backend.Deployer.finish(Deployer.java:572)
com.sun.enterprise.deployment.backend.ModuleDeployer.doRequestFinish(ModuleDeployer.java:226)
com.sun.enterprise.deployment.phasing.J2EECPhase.runPhase(J2EECPhase.java:208)
com.sun.enterprise.deployment.phasing.DeploymentPhase.executePhase(DeploymentPhase.java:108)
com.sun.enterprise.deployment.phasing.PEDeploymentService.executePhases(PEDeploymentService.java:966)
com.sun.enterprise.deployment.phasing.PEDeploymentService.deploy(PEDeploymentService.java:283)
com.sun.enterprise.deployment.phasing.PEDeploymentService.deploy(PEDeploymentService.java:835)
com.sun.enterprise.management.deploy.DeployThread.deploy(DeployThread.java:187)
com.sun.enterprise.management.deploy.DeployThread.run(DeployThread.java:225)
Parent -> ASChain parentCL :: Addons Chain parentCL :: Shared ClassLoader Chain parentCL :: sun.misc.Launcher$AppClassLoader@6d6f0472 URLs ::
, file:/Users/neilbeveridge/Library/glassfish/lib/javaee.jar, file:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/tools.jar, file:/Users/neilbeveridge/Library/glassfish/lib/install/applications/jmsra/imqjmsra.jar, file:/Users/neilbeveridge/Library/glassfish/lib/com-sun-commons-launcher.jar, file:/Users/neilbeveridge/Library/glassfish/lib/com-sun-commons-logging.jar, file:/Users/neilbeveridge/Library/glassfish/imq/lib/jaxm-api.jar, file:/Users/neilbeveridge/Library/glassfish/imq/lib/fscontext.jar, file:/Users/neilbeveridge/Library/glassfish/imq/lib/imqbroker.jar, file:/Users/neilbeveridge/Library/glassfish/imq/lib/imqjmx.jar, file:/Users/neilbeveridge/Library/glassfish/imq/lib/imqxm.jar, file:/Users/neilbeveridge/Library/glassfish/lib/webservices-rt.jar, file:/Users/neilbeveridge/Library/glassfish/lib/webservices-tools.jar, file:/Users/neilbeveridge/Library/glassfish/lib/mail.jar, file:/Users/neilbeveridge/Library/glassfish/lib/appserv-jstl.jar, file:/Users/neilbeveridge/Library/glassfish/lib/jmxremote_optional.jar, file:/Users/neilbeveridge/Library/glassfish/lib/SUNWjdmk/5.1/lib/jdmkrt.jar, file:/Users/neilbeveridge/Library/glassfish/lib/activation.jar, file:/Users/neilbeveridge/Library/glassfish/lib/appserv-rt.jar, file:/Users/neilbeveridge/Library/glassfish/lib/appserv-admin.jar, file:/Users/neilbeveridge/Library/glassfish/lib/appserv-cmp.jar, file:/Users/neilbeveridge/Library/glassfish/updatecenter/lib/updatecenter.jar, file:/Users/neilbeveridge/Library/glassfish/jbi/lib/jbi.jar, file:/Users/neilbeveridge/Library/glassfish/imq/lib/imqjmx.jar, file:/Users/neilbeveridge/Library/glassfish/lib/ant/lib/ant.jar, file:/Users/neilbeveridge/Library/glassfish/lib/dbschema.jar, file:/Users/neilbeveridge/Library/glassfish/lib/appserv-deployment-client.jar, file:/Users/neilbeveridge/Library/glassfish/lib/appserv-ee.jar, file:/Users/neilbeveridge/Library/glassfish/lib/appserv-ext.jar, file:/Users/neilbeveridge/Library/glassfish/lib/appserv-jwsacc.jar, file:/Users/neilbeveridge/Library/glassfish/lib/appserv-launch.jar, file:/Users/neilbeveridge/Library/glassfish/lib/appserv-se.jar, file:/Users/neilbeveridge/Library/glassfish/lib/appserv-tags.jar, file:/Users/neilbeveridge/Library/glassfish/lib/appserv-ws.jar, file:/Users/neilbeveridge/Library/glassfish/lib/asm-3.1.jar, file:/Users/neilbeveridge/Library/glassfish/lib/j2ee.jar, file:/Users/neilbeveridge/Library/glassfish/lib/jackson-asl-0.9.4.jar, file:/Users/neilbeveridge/Library/glassfish/lib/jersey-bundle-1.0.3.1.jar, file:/Users/neilbeveridge/Library/glassfish/lib/jesmf-plugin.jar, file:/Users/neilbeveridge/Library/glassfish/lib/jettison-1.0.1.jar, file:/Users/neilbeveridge/Library/glassfish/lib/jhall.jar, file:/Users/neilbeveridge/Library/glassfish/lib/jmac-api.jar, file:/Users/neilbeveridge/Library/glassfish/lib/jsf-impl.jar, file:/Users/neilbeveridge/Library/glassfish/lib/jsr311-api-1.0.jar, file:/Users/neilbeveridge/Library/glassfish/lib/jxta.jar, file:/Users/neilbeveridge/Library/glassfish/lib/Pack200Task.jar, file:/Users/neilbeveridge/Library/glassfish/lib/shoal-gms.jar, file:/Users/neilbeveridge/Library/glassfish/lib/toplink-essentials-agent.jar, file:/Users/neilbeveridge/Library/glassfish/lib/toplink-essentials.jar, file:/Users/neilbeveridge/Library/glassfish/
constituent CLs ::
constituent CLs ::
:: asimpl :: optionalChain
was requested to find class com.sun.enterprise.management.deploy.DeployThread.LogStrings_en after done was invoked from the following stack trace
java.lang.Throwable
at com.sun.enterprise.loader.EJBClassLoader.findClassData(EJBClassLoader.java:708)
at com.sun.enterprise.loader.EJBClassLoader.findClass(EJBClassLoader.java:628)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:296)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2289)
at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1364)
at java.util.ResourceBundle.findBundle(ResourceBundle.java:1328)
at java.util.ResourceBundle.findBundle(ResourceBundle.java:1282)
at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1224)
at java.util.ResourceBundle.getBundle(ResourceBundle.java:952)
at java.util.logging.Logger.findResourceBundle(Logger.java:1257)
at java.util.logging.Logger.setupResourceInfo(Logger.java:1312)
at java.util.logging.Logger.getLogger(Logger.java:312)
at com.sun.enterprise.server.logging.BaseLogManager.doInitializeLogger(BaseLogManager.java:125)
at com.sun.enterprise.server.logging.BaseLogManager.addLogger(BaseLogManager.java:196)
at java.util.logging.LogManager.demandLogger(LogManager.java:393)
at java.util.logging.Logger.getLogger(Logger.java:274)
at com.sun.enterprise.management.deploy.DeployThread.trace(DeployThread.java:145)
at com.sun.enterprise.management.deploy.DeployThread.run(DeployThread.java:226)


====

After some searching I tracked the problem down to a syntax error in sun-web.xml. In my particular case I was using the following construct:

<property name="alternatedocroot_1" value="from=/resources/template/* dir=/Users/neilbeveridge/temp/extroot/admin"/>

I had mistakenly inserted an opening " following the dir sub-attribute. In this case the whole from=* dir=* construct must go inside the value attribute in XML.