Exporting H.264 from Cinelerra

Tags

This article will discuss creation of H.264 videos from Cinelerra on Ubuntu 12.04. A lot of the information was synthesized from
Olson Video and Websites.

Built-in H.264 Export

In the Renderer select “Quicktime for Linux”, and then use the H.264 compression option for video, and MPEG-4 Audio for sound:

01 H.264 - built-in

The problem with this approach is that Cinelerra provides very few settings to customize the encoding. I can achieve a much higher quality for the same size by using 2-pass encoding with the x264 utility. Luckily for us, Cinelerra has YUV4MPEG Stream option.

YUV4MPEG Stream

YUV4MPEG Stream allows us to pipe the video stream out of Cinelerra into an arbitrary tool chain. In this case the tool chain is my script that uses named pipes to encode the video with x264.

The script was adopted from here: http://www.renomath.org/ejolson/video/hddvd/264wf.html
I used http://www.renomath.org/ejolson/video/hddvd/pipe-x264 as a starting point.

Preparation

  1. Make sure that you have the x264 package installed.
  2. Save the code provided in the last section of this post as /usr/local/bin/pipe-x264.sh
  3. Make /usr/local/bin/pipe-x264.sh executable.
  4. Specify the desired resolution and FPS in the script. See code in the last section, which shows you which lines to modify.
  5. Cinelerra must be launched from the command line, so that you can see the console output.

Export Steps

1) Export The Sound

In the Renderer select “Microsoft WAV” for File Format, check Audio and uncheck Video. Export.

2) First Pass Video

In the Renderer select “YUV4MPEG Stream” for File Format, check Video and uncheck Audio. In the settings check “Use Pipe” and type the following:

$ /usr/local/bin/pipe-x264.sh -p 1 -b 1000 "%"

The first parameter (-p) is the pass number. The second (-b) is the bitrate you want. Make sure that the path points to where you’ve saved the script, that I’ll give you below.

If all works well, the video should start encoding. To check if it is actually working, you must launch Cinelerra from the command line and look at the console output after you start rendering. You should see x264 command call printed and x264 should be running and printing stuff to the console. If something went wrong, you’ll see an error instead. If you can’t figure out what’s wrong from the output, leave a comment and I might try to help.

The output in the console should look something like this:

Running pass 1:
     x264 /tmp/cine_pipe --input-res 1920x1080 --fps 59.9401 --bitrate 1000     --pass 1 --stats "/mnt/hde/temp/work/gym_2013-03-07/test40.stats"     --bframes 2 --b-adapt 2     --direct auto     --threads auto     --output "/mnt/hde/temp/work/gym_2013-03-07/test40.m2v"

[yuv4mpegpipe @ 0x15ae720] Estimating duration from bitrate, this may be inaccurate
lavf [info]: 1920x1080p 0:1 @ 60000/1001 fps (cfr)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2
x264 [info]: profile Main, level 4.2
Render::render_single: Session finished.
x264 [info]: frame I:1     Avg QP:43.42  size: 10362
x264 [info]: frame P:49    Avg QP:39.33  size:  3682
x264 [info]: frame B:95    Avg QP:41.30  size:   788
x264 [info]: consecutive B-frames:  0.7%  4.1% 95.2%
x264 [info]: mb I  I16..4: 97.9%  0.0%  2.1%
x264 [info]: mb P  I16..4: 11.4%  0.0%  0.0%  P16..4:  9.3%  0.0%  0.0%  0.0%  0.0%    skip:79.2%
x264 [info]: mb B  I16..4:  0.7%  0.0%  0.0%  B16..8:  2.8%  0.0%  0.0%  direct: 1.0%  skip:95.6%  L0:39.7% L1:59.7% BI: 0.6%
x264 [info]: final ratefactor: 34.33
x264 [info]: direct mvs  spatial:88.4% temporal:11.6%
x264 [info]: coded y,uvDC,uvAC intra: 1.7% 25.6% 1.4% inter: 0.2% 1.5% 0.0%
x264 [info]: i16 v,h,dc,p: 46% 31% 16%  7%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 16% 25% 36%  2%  3%  6% 10%  2%  1%
x264 [info]: i8c dc,h,v,p: 76% 12% 11%  2%
x264 [info]: Weighted P-Frames: Y:0.0% UV:0.0%
x264 [info]: kb/s:878.42

3) Second Pass Video

Once the first pass render is finished, go back to Video Settings in the Renderer and change the pipe command to run the second pass (-p 2):

$ /usr/local/bin/pipe-x264.sh -p 2 -b 1000 "%"

Again check the console to make sure that encoding has started.

4) Multiplex Video and Sound

The previous steps should have produced two files: test40.m2v and test40.wav. We need to multiplex them into a container:

$ ffmpeg -i test40.m2v -i test40.wav -acodec libmp3lame -ab 128k -ar 48000 -vcodec copy test40.avi

Here is a handy script to keep around for this step:

#!/bin/bash
# Multiplex video and sound
#

if [[ $# -lt 2 ]] ; then
    echo
    echo "  Usage: $0 <video file> <sound file>"
    echo
    exit 1
fi

command="ffmpeg -i $1 -i $2 -acodec libmp3lame -ab 128k -ar 48000 -vcodec copy ${1%%.*}.avi"

echo $command
eval $command

Mission Accomplished

You should now have a nice H.264 avi file.

Note that the .m2v file will probably not play correctly in any player, but that is ok, b/c after we multiplex it with sound the resulting file plays everywhere.

pipe-x264.sh

Please note that you have to set the desired resolution and framerate on the lines highlighted below.

#!/bin/bash
#
# Use this script to export H.264 videos from Cinelerra
#

# Change these to suite your needs
######## Canon G6 ##############
#frame_rate=10
#resolution=640x480
######## Sony A57 ##############
frame_rate=59.9401
resolution=1920x1080

pass=1
bit_rate=10000

help(){
    cat <<END
Usage: $0 [options] filename.anything
Options:
    -b n      bitrate n                             ($bit_rate)
    -p n      pass n                                ($pass)
    -h        Print this help message
END
    exit 0
}

while getopts b:p:h name "$@"
do
    case $name in
b)
	bit_rate=$OPTARG ;;
p)
	pass=$OPTARG ;;
*)
    help ;;
    esac
done
let shiftind=$OPTIND-1
shift $shiftind
if test "$#" != "1"
then
    help
fi

outfile=$1
base=`echo $outfile | sed "s/\.[^.]*$//"`

command="x264 /tmp/cine_pipe --input-res $resolution --fps $frame_rate --bitrate $bit_rate \
    --pass $pass --stats \"$base.stats\" \
    --bframes 2 --b-adapt 2 \
    --direct auto \
    --threads auto \
    --output \"$outfile\""

# Make a named pipe
rm /tmp/cine_pipe 2> /dev/null
mkfifo /tmp/cine_pipe

echo "Running pass $pass:"
echo "     $command"
echo

# Run the encoding command. It will block and wait for cat to start feeding data into the pipe
eval "$command &"

cat > /tmp/cine_pipe

Hope this helps.

Installing Cinelerra 4.4 on Ubuntu 12.04

This article provides step-by-step instructions for installing Cinelerra 4.4.

I upgraded my main workstation to Ubuntu 12.04 recently, briefly looked at PiTiVi and decided that Cinelerra is still the only capable video editor for Linux, despite its many flaws.

This post provides step-by-step instructions for compiling and installing Cinelerra 4.4.

Install Dependencies

Here is the list of dependencies that I know about. At a minimum you’ll need these packages, although there may be more. If the compile fails on a dependency, simply look at the last few lines of the output and the error should tell you what it’s missing.

$ sudo apt-get install wget bzip2 patch build-essential w32codecs w64codecs libtool nasm libncurses5-dev libbz2-dev libncursesw5-dev libxv-dev libxxf86vm-dev libogg-dev libvorbis-dev libtheora-dev libopenexr-dev libdv-dev libpng-dev libjpeg62-dev libfreetype6-dev libfaad-dev libsndfile1-dev uuid-dev libavutil-dev libmpeg3-dev libavcodec-dev libx264-dev libfaac-dev libglu1-mesa-dev libmjpegtools-dev

Download Source

$ wget http://downloads.sourceforge.net/project/heroines/cinelerra-4.4-src.tar.xz
$ tar xvf cinelerra-4.4-src.tar.xz

You should now have a directory called cinelerra-4.4

Apply Patches

You will need to apply two patches from the Community Version of Cinelerra to add dnxhd and dv50 codec support and YUV Stream support. This was easy, b/c the patches applied cleanly.

$ wget http://renomath.org/video/linux/cinelerra/patch-cinelerra-4.4-dnxhd
$ wget http://renomath.org/video/linux/cinelerra/patch-cinelerra-4.4-yuvstream
$ patch -Np1 -d cinelerra-4.4 < patch-cinelerra-4.4-dnxhd
$ patch -Np1 -d cinelerra-4.4 < patch-cinelerra-4.4-yuvstream

I found out about these patches from Eric Olson’s site: http://renomath.org/video/linux/cinelerra/

Build

$ cd cinelerra-4.4
$ ./configure
$ make

The install target does not work, so don’t bother with it. After make succeeds, the Cinelerra binary will be in the bin/ folder. So, to run Cinelerra:

$ cd bin
# ./cinelerra

Importing AVCHD

I mostly use Cinelerra to work with AVCHD video from my Sony DSLR, which shoots at 1080p with 60 FPS. Cinelerra cannot read the resulting MTS files, so I have to convert them to dnxhd (AVdn) format first:

$ ffmpeg -i 00014-360_2.MTS -b 185M -vcodec dnxhd -acodec pcm_s16le -threads 4 output.mov

Also, I could not get Cinelerra to read the sound from the resulting file (complains about missing ‘lpcm’ codec), so I had to take another intermediate step to extract the audio from the file, so I can import it separately:

$ mplayer output.mov -vo null -vc dummy -ao:file=sound.wav

Once I import output.mov into Cinelerra, I delete the audio tracks and I replace them with sound.wav, which work well.

Improvements

Aside from some new features here and there, the most obvious advantage of Cinelerra 4.4 over the Community Version is that the video processing and playback appears to be faster. I get about 10 FPS in cinelerra-cv, whereas Cinelerra 4.4 gives me 20 FPS, which makes the video manipulation and playback much smoother. Nice.

Missing Faders in Video Tracks

Aside from some minor problems, like missing icons for transitions and other resources, the most severe problem I ran into was the missing Fader line in video tracks. The audio tracks still had the Faders, but in Video tracks the horizontal white line was just gone.

Here is what it looks like:

Missing Faders

If this happens to you, the fix is to press the 3rd button from the right in the tool bar. It is called “Fit autos to display (Alt + f)”. What it will do is change the fade range in the drop down at the bottom from “-10.00 – 10.00” to something like “-33.00 – 133.00” in my case. You can also select “0 – 100” from that drop down as an alternative.

Navigate <rich:tabPanel /> with keyboard

<rich:tabPanel /> does not support any keyboard interactions out of the box. One way to add it is to include the following javascript code:

 

/**
 * WCAG requires keyboard navigation for tabs. This code turns all inactive tabs into links,
 * so the browser starts tabbing over them. 
 * 
 * Author: Val Blant
 */
jQuery(document).ready(function() {
	jQuery("td.rf-tab-hdr-inact").children('span').each(function() {
		var tabName = jQuery(this).html();
		jQuery(this).empty();
		jQuery(this).append("<a href='#' style='text-decoration:none; color: black;' onclick='return false;'>" + tabName + "</a>");
	});
});

Eclipse Open Type dialog hangs

Tags

I noticed that on recent versions of Eclipse and Ubuntu, Eclipse’s Open Type dialog (Ctrl-Shift-T) often freezes for an annoying period of time as soon as you start typing your class name.

Here is the related bug:
https://bugzilla.gnome.org/show_bug.cgi?id=575873

Workaround

A temporary solution I found is to make sure that gail is not used by simply renaming the file:

/usr/lib/x86_64-linux-gnu:
lrwxrwxrwx 1 root root 22 Feb 4 16:18 libgailutil-3.so.0 -> libgailutil-3.so.0.0.0
-rw-r--r-- 1 root root 35440 Apr 16 2012 libgailutil-3.so.0.0.0
-rw-r--r-- 1 root root 31304 Mar 26 2012 libgailutil.so.18.0.1.OFF

Triggering a “change” ajax event on <rich:calendar />.

Problem Statement

Let’s say that we want to trigger an ajax request when the user changes the value in a <rich:calendar />, but it has to work when the user types the date into the text field manually, as well as when the user uses the calendar to pick the date.

<rich:calendar /> defines two events just for this purpose:

  • “change”: Triggered when the date is changed from the calendar
  • “inputchange”: Triggered when the text in the text field is changed manually

But how do we combine them?

Solution

I already had my calendar inside a Composite JSF Component, so my solution uses the following approach.

YourWebApp/WebContent/resources/components/calendar.xhtml:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" 
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

<html xmlns="http://www.w3.org/1999/xhtml"
	xmlns:h="http://java.sun.com/jsf/html"
	xmlns:f="http://java.sun.com/jsf/core"
	xmlns:c="http://java.sun.com/jsp/jstl/core"
	xmlns:rich="http://richfaces.org/rich"
	xmlns:a4j="http://richfaces.org/a4j"
	xmlns:composite="http://java.sun.com/jsf/composite">

<composite:interface>
	<composite:attribute name="value" required="true" />
	<composite:attribute name="timeZone" required="true" />
	<composite:attribute name="disabled" default="false" />
	<composite:attribute name="displayValueOnly" default="false" />
	<composite:attribute name="required" />

	<composite:clientBehavior name="date_change" event="change" targets="#{cc.id}"/>
	<composite:clientBehavior name="date_change" event="inputchange" targets="#{cc.id}"/>
</composite:interface>

<composite:implementation>

	<c:if test="#{cc.attrs.displayValueOnly == false}">
		<rich:calendar id="#{cc.id}" 
				inputSize="10" 
				required="#{cc.attrs.required}" 
				enableManualInput="true" 
				value="#{cc.attrs.value}" 
				disabled="#{cc.attrs.disabled}" 
				datePattern="yyyy-MM-dd"
				onchange="if (DirtyFieldHandler) DirtyFieldHandler.setDirty();">
			<f:convertDateTime pattern="yyyy-MM-dd" timeZone="#{cc.attrs.timeZone}" />
		</rich:calendar>
	</c:if>

	<c:if test="#{cc.attrs.displayValueOnly == true}">
		<h:inputText value="#{cc.attrs.value}" disabled="true" id="#{cc.id}InputDate" size="10">
			<f:convertDateTime pattern="yyyy-MM-dd" timeZone="#{cc.attrs.timeZone}" />
		</h:inputText>
	</c:if>

</composite:implementation>
</html>

Then we use the component like so:

  <comp:calendar value="#{backingBean.approvedDate}">
     <a4j:ajax event="date_change" execute="@this" />
  </comp:calendar>

“Error: During update” from <rich:dataTable /> on ajax footer refresh

Problem Statement

Consider the following test code:

<rich:dataTable id="testTable" 
	value="#{dataTableTestBackingBean.rowsModel}"
	var="rowVar">
	
	<f:facet name="header">
	   <rich:column>
		<h:outputText value="Column 1" />
   	   </rich:column>
	</f:facet>
	
	<rich:column>
	   <h:inputText id="valTest" value="#{rowVar.col1}" >
       	 	<a4j:ajax 
       	 		event="blur" 
       	 		render="testColumn, footerTest" 
       	 		limitRender="true" 
       	 		execute="@this" />
		</h:inputText>
	</rich:column>
	<rich:column>
	   <h:outputText id="testColumn" value="#{dataTableTestBackingBean.footerValue}" />
	</rich:column>




	<f:facet name="footer">
	   <rich:column>
		<h:outputText id="footerTest" value="#{dataTableTestBackingBean.footerValue}" />
	   </rich:column>
	</f:facet>
</rich:dataTable>

This example will fail with the following Javascript error in the browser:

Error: During update: formId:testTable:0:footerTest not found

This is expected, b/c the correct id for the footer is actually formId:testTable:footerTest.

The bug is in org.richfaces.context.ComponentIdResolver. Here is what happens:

  1. The RenderComponentCallback is executed on the input component (‘valTest’ in the example) and it reads the list of short ids to render from the attached Ajax Behavior. Note that we got here by walking the parent data table model, so the current rowKey is 0.
  2. RenderComponentCallback asks ComponentIdResolver to resolve short ids into client IDs
  3. ComponentIdResolver starts ascending up the tree from ‘valTest’ and looking at facets and children until it finds ‘footerTest’.
  4. At this point it asks for the clientId of ‘footerTest’ w/o regard for the fact that the data model has a rowKey that is set 0

So, we get the wrong id b/c we call UiData.getClientId() outside of the normal walking order of that model.

I have logged a bug here: https://issues.jboss.org/browse/RF-11134

Workaround

I noticed that if I nest the above data table in another data iterator, then the footer update suddenly starts working. I investigated and it turned out that nested table will have getClientId() called on all of its components by the parent table, and since clientIds are cached, by the time ComponentIdResolver gets to it, it will use the cached id, rather than looking it up at the wrong time. Non-nested tables simply don’t get around to calling getClientId() on their components before ComponentIdResolver runs, which is probably the real problem.

Client-side Fix

Until the problem is fixed in RichFaces, the following client-side code fixes the problem. The idea here is to intercept JSF Response handler calls and check if all IDs in the responseXML are in fact present in the page. If they are not, use the simple ID to find the real clientId and correct it in the response. After that is done, control is returned to the usual RF Ajax handling code.

YourWebApp/WebContent/resources/javascript/richfaces-ajax-fix.js:

/**
 * Temporary fix for https://issues.jboss.org/browse/RF-11134
 * 
 * Once the ComponentIdResolver or UIDataAdapter.saveChildState is fixed,
 * this file will no longer be needed.
 * 
 * @author Val Blant
 */

if (!window.RichFacesAjaxFix) {
	window.RichFacesAjaxFix = {};
}


if ( typeof jsf != 'undefined' ) {

(function($, raf, jsf) {
	
	raf.ajaxContainer = raf.ajaxContainer || {};
	
	if (raf.ajaxContainer.jsfResponse) {
		return;
	}

	/** 
	 * Save the original JSF 2.0 (or Richfaces) method 
	 */
	raf.ajaxContainer.jsfResponse = jsf.ajax.response;

	/**
	 * Check if all IDs in the responseXML are in fact present in the page. 
	 * If they are not, use the simple ID to find the real clientId.
	 */
	jsf.ajax.response = function(request, context) {
		
		var changes = raf.ajaxContainer.extractChanges(request);
		
		for (var i = 0; i < changes.length; i++) {
			var change = changes[i];

			var oldClientId = raf.ajaxContainer.getClientId(change);
			if ( oldClientId == null ) {
				// This is probably not an "update" change.
				continue;
			}
			var shortId = raf.ajaxContainer.getShortId(change);
			

            // Check if this target element actually exists. If not, then we found the bug
			//
            var target = document.getElementById(oldClientId);
            if ( !target ) {
            	
	            // Find the target by shortId
            	targets = jQuery("[id$='" + shortId + "']");
            	
            	if ( targets.length === 0 ) {
            		// ok.... this target is not on the page at all. Eject.
            		continue;
            	}
            	
            	if ( targets.length === 1 ) {
            		// Only one target found, so we can derive the exact clientId easily
            		var newClientId = targets.first().attr("id");
            	}
            	else {
            		// This shouldn't happen, b/c nested data tables do not have this bug
            		//
            		// However.... looks like this can happen if we find an element with the same
            		// short id on another tab.
            		// We pick the right one with a VERY BADLY CODED approach - we choose the target with 
            		// the id whose length is closest to the original clientId length, b/c they should differ in just
            		// the row index. It's unreliable, but since this is a temporary solution, I'm not spending time to
            		// code this properly.
            		//
            		var smallest_length_delta = 1000;
            		var chosenidx = -1;
            		for (var k = 0; k < targets.length; k++) {
            			var length_delta = oldClientId.length - targets.get(k).id.length ;
            			if ( length_delta < smallest_length_delta ) {
            				chosenidx = k;
            				smallest_length_delta = length_delta;
            			}
            		}
            		
            		var newClientId = targets.get(chosenidx).id;
            	}
            	
            	// Now we have the clientId, so we just need to replace it
            	// in the ID attribute and in the markup itself
            	//
            	if ( newClientId ) {
            		raf.ajaxContainer.fixChange(change, oldClientId, newClientId);
            	}
            }
		}

		// Delegate to the old code
		//
		raf.ajaxContainer.jsfResponse(request, context);

	};
	
	/**
	 * Returns the list of elements under the 'changes' element in the AJAX response
	 */
	raf.ajaxContainer.extractChanges = function(request) {
		var changes = [];
		var xml = request.responseXML;
		
		if (xml !== null) {
			var pr = xml.getElementsByTagName("partial-response");
			
			if ( pr && pr.length > 0 ) {
				var responseType = pr[0].firstChild;
				
				if (responseType && responseType.nodeName === "changes") {
					changes = responseType.childNodes;
				}
			}
		}
		
		return changes;
	};
	
	raf.ajaxContainer.getClientId = function(change) {
		return change.getAttribute('id');
	};

	/**
	 * Return the id part after the last ':' 
	 */
	raf.ajaxContainer.getShortId = function(change) {
		var clientId = raf.ajaxContainer.getClientId(change);
		var ncs = clientId.split(":");
		var shortId = ncs[ncs.length - 1];
		
		return shortId;
	};

	/**
	 * Update the id of the given change element and the content with the newClientId
	 */
	raf.ajaxContainer.fixChange = function(change, oldClientId, newClientId) {
    	change.setAttribute("id", newClientId);
    	
    	for (var i = 0; i < change.childNodes.length; i++) {
            node = change.childNodes[i];
            node.nodeValue = node.nodeValue.replace(oldClientId, newClientId);
        }
	};


}(jQuery, window.RichFacesAjaxFix, jsf));
}

Dirty <rich:tabPanel />

Tags

,

I have recently had to implement a new requirement for all pages in our app that have a RichFaces Tab Panel. This post is about RichFaces 4.2.Final.

Requirement

Change tabs with Unsaved Changes:
1. Issue the following warning:

There are pending changes to the current tab that have not been saved. Do you wish to proceed with the tab change, or cancel and return to the tab?

2. The User may elect to either:
2.1. Proceed; The changes to the current tab are discarded.
2.2. Cancel; The use case continues at the step at which the user elected to change tabs.

Analysis

rich:tabPanel provides us with the following tools:

  • We can call a server method when the tab is changed (if we use ajax), but w/o ability to stop the tab change or show any warnings
  • We can call arbitrary javascript with ability to warn the user and abort tab switch, but the javascript is rendered at page render time, not at tab switch time, so no info from the server can be communicated

I did not want to get into server side tracking of all changes, since that would really complicate all pages, so the solution must be client side only.

Solution

I have created a very simple framework that I call Dirty Field Handler. It works as follows:

  • dirtyFieldHandler.js attaches a “delegated” event handler to the document, which will handle all “change” events from any input field on the page
  • Developer can use the DirtyFieldHandler framework as follows:
<!-- Include the framework on the page that needs it -->
 	<h:outputScript library="javascript" name="dirtyFieldHandler.js" target="head"/>

 	.....

 	<rich:tabPanel
 		switchType="ajax" <!-- Can't call the server w/o using ajax tab switches! -->
 		onbeforeitemchange='return DirtyFieldHandler.warnUser("#{msgs["common.warn.unsavedTab"]}");' <!-- Smart Javascript popup -->
 		itemChangeListener="#{initiativeBackingBean.tabSwitchAction}"> <!-- Notify the server that tab was switched -->

 		......

	</rich:tabPanel>

Code

 
dirtyFieldHandler.js:

/**
 * Provides the ability to detect if any input fields on the page were dirtied.
 * 
 * Exposes DirtyFieldHandler.warnUser(message, [true/false]) function that can be called
 * by the page to warn the user about dirty fields.
 * 
 * @author Val Blant
 */

if (!window.DirtyFieldHandler) {
	window.DirtyFieldHandler = {};
}


(function($, dfh) {
	
	dfh.setDirty = function() {
		dfh.dirty = true;
	}

	dfh.resetDirty = function() {
		dfh.dirty = false;
	}

	/**
	 * Warns the user if any fields are dirty. If the user chooses
	 * to navigate away, clears the dirty flag, unless the 2nd parameter is used
	 * to override that behavior
	 */
	dfh.warnUser = function(event, message, clearDirty) {
		var leavePage = true;
		clearDirty = typeof clearDirty !== 'undefined' ? clearDirty : true;
		
		if (dfh.dirty) {
			leavePage = confirm(message);
			if ( leavePage && clearDirty) {
				dfh.resetDirty();
			}
		}

		return leavePage;
	}
	
	
	
	// Init dirty flag
	//
	dfh.resetDirty();
	
	// Attach a "delegated" event handler to the document. All events that bubble
	// from input elements under this document will execute the handler.
	// We use a delegated handler b/c this approach will detect input field
	// changed or added by ajax requests
	//
	$(document).on("change", ":input", dfh.setDirty)

}(jQuery, window.DirtyFieldHandler));

Taming Weblogic

Tags

,

When things go bad in Weblogic, finding the problem can be extremely tricky. This post outlines some troubleshooting techniques that can save you some serious amounts of time and frustration.

Introduction

In order to deploy an application on WebLogic (WLS from here on), you must first create a domain. You can do this by running:

val@thinkpad:/opt/Oracle/Middleware/wlserver_10.3/common/bin$ ./config.sh

After that you need to create a “Machine” (Environment -> Machines) and point it to a running instance of a Node Manager. Then you create a Server (Environment -> Servers) and point it to the Machine. Then you can specify Deployments (EARs, WARS, etc) and run them on a specific server.

Deployments can either be archive files, or an exploded version of an archive on the file system. This is especially useful if you are tweaking your deployment by making modifications to it from Eclipse.

For example, let’s say you wanted to make some changes and redeploy the EAR. The steps would be:

  1. Stop your Server: Environments -> Servers -> Example_Server -> Control -> Shutdown
  2. Make the changes in Eclipse
  3. Export your ear or weblogic jar. The structure of a Weblogic application is explained in confusing detail here: http://docs.oracle.com/cd/E13222_01/wls/docs81/programming/environment.html
  4. Go to http://localhost:7001/console/ -> Deployments -> Update. This step assumes that you have previously deployed the app.
  5. Start the server

Starting Weblogic

val@thinkpad:/opt/Oracle/Middleware/wlserver_10.3/server/bin$ ./setWLSEnv.sh
val@thinkpad:/opt/Oracle/Middleware/wlserver_10.3/server/bin$ ./startNodeManager.sh
val@thinkpad:/opt/Oracle/Middleware/user_projects/domains/base_domain$ ./startWebLogic.sh

Changing Java interpreter

/opt/Oracle/Middleware/wlserver_10.3/common/bin/commEnv.sh:

@@ -129,7 +129,7 @@ if [ -z "${JAVA_HOME}" -o -z "${JAVA_VENDOR}" ]; then
   JAVA_HOME="/opt/Oracle/Middleware/jrockit_160_14_R27.6.5-32"
   # Set up JAVA VENDOR, possible values are
   #Oracle, HP, IBM, Sun ...
-  JAVA_VENDOR=Oracle
+  JAVA_VENDOR=Sun
   # PRODUCTION_MODE, default to the development mode
   PRODUCTION_MODE=""
 fi

Tweaking VM memory arguments

Go to /opt/Oracle/Middleware/user_projects/domains/base_domain/bin/setDomainEnv.sh and find the section for the chosen jdk and architecture. PermGen options are in the same file.

Logs

The logs you will be interested in are located under the server directory. For example:

/opt/Oracle/Middleware/user_projects/domains/base_domain/servers/Example_Server/logs

There will be .log files, which are less verbose and .out files, which is what you’ll want most of the time.

By default WLS is configured to provide as little information as possible in those logs, which is why it is unlikely that you’ll be able to fix your problem with default configuration. However, it is possible to make those logs much more verbose.

Debug Logging

-Dweblogic.StdoutDebugEnabled=true 
-Dweblogic.log.RedirectStdoutToServerLogEnabled=true

Classloader Logging

If you are dealing with NoClassDefFoundError, or other classloading issues, turn on classloader logging:

-Dweblogic.utils.classloaders.ClasspathClassFinder=true 
-Dweblogic.utils.classloaders.GenericClassLoader.Verbose=true 
-Dweblogic.utils.classloaders.ChangeAwareClassLoader.Verbose=true

General Debug Flags

-Dlog4j.debug=true
-Dweblogic.debug.DebugConfigurationEdit=true
-Dweblogic.debug.DebugDeploymentTaskRuntime=true
-Dweblogic.debug.DebugDeploymentManagerAdmin=true
-Dweblogic.debug.DebugDeploymentManagerTarget=true
-Dweblogic.debug.DebugDeploymentOperationsAdmin=true
-Dweblogic.debug.DebugDeploymentManagerTargetOperations=true
-Dweblogic.debug.DebugDeploymentServiceApiTargetCalls=true
-Dweblogic.debug.DebugDeploymentServiceApiAdminCalls=true
-Dweblogic.debug.DebugDeploymentServiceApiAdminCallback=true
-Dweblogic.debug.DebugDeploymentServiceApiTargetCallback=true
-Dweblogic.debug.DebugDeploymentServiceStatusUpdatesAdmin=true
-Dweblogic.debug.DebugDeploymentServiceTransport=true
-Dweblogic.debug.DebugDeploymentServiceStatusUpdatesTarget=true

Even more debug flags

http://middlewaremagic.com/weblogic/?page_id=1096

Configuring Logging

These settings need to be passed in as VM arguments to the server JVM. This is done here:

Environment -> Servers -> Example_Server -> Server Start -> Arguments

For example, my Arguments field looks like this right now:

-Dweblogic.StdoutDebugEnabled=true -Dweblogic.log.RedirectStdoutToServerLogEnabled=true -Xms512m -Xmx1024m -XX:MaxPermSize=512m

Debugging with Eclipse

If increased logging did not help you, it’s time to get serious :). We’ll connect to the running WLS JVM with the Eclipse Debugger and debug the internals of WLS in order to find the problem.

  • Make sure that JadClipse plugin is installed and working properly.
  • Add the following parameters to Server -> Server Start -> Arguments:
-Xdebug -Xrunjdwp:transport=dt_socket,address=1044,server=y,suspend=y

suspend=y means that when you start the server JVM, it will immediately get suspended and wait for a debugger to attach.

You might also want to add other VM arguments during this process. For example, my Arguments field looks like this:

-Xdebug -Xrunjdwp:transport=dt_socket,address=1044,server=y,suspend=y -Dweblogic.utils.classloaders.ClasspathClassFinder=true -Dweblogic.utils.classloaders.GenericClassLoader.Verbose=true -Dweblogic.utils.classloaders.ChangeAwareClassLoader.Verbose=true -Dweblogic.StdoutDebugEnabled=true -Dweblogic.log.RedirectStdoutToServerLogEnabled=true -Xms512m -Xmx1024m -XX:MaxPermSize=512m
  • Go to Eclipse, open your project’s Build Path and include /opt/Oracle/Middleware/wlserver_10.3/server/lib/weblogic.jar.
  • Create a Remote Java Application debug launcher. Address is localhost:1044
  • Start the server from the WLS Admin console. The status of the server will change to STARTING and nothing else will happen, b/c the server process is now suspended.
  • Run the Remote Java Application debug launcher you created earlier. The WLS server will continue booting and you should now be able to place breakpoints into WebLogic code!

That’s it – happy debugging!

Primefaces and Broken Head Resource Ordering

Tags

, ,

I have recently tried using PrimeFaces with a RichFaces application. I am very impressed with PrimeFaces. It is a well designed, simple, yet powerful JSF framework with a lot of cool components. Unfortunately, it does have bugs that I found as soon as I tried to use it.

Introduction

PrimeFaces is an open source JSF component suite with various extensions.

  • Rich set of components (HtmlEditor, Dialog, AutoComplete, Charts and many more).
  • Built-in Ajax based on standard JSF 2.0 Ajax APIs.
  • Lightweight, one jar, zero-configuration and no required dependencies.
  • Ajax Push support via websockets.
  • Mobile UI kit to create mobile web applications for handheld devices.
  • Skinning Framework with 35+ built-in themes and support for visual theme designer tool.
  • Extensive, clear documentation.
  • Large, vibrant and active user community.

Interaction with JSF stack

PrimeFaces uses the following classes to insert itself into the JSF framework. i.e – these are the classes which will be doing stuff to our requests, even if we are not using any PrimeFaces components!

  • org.primefaces.context.PrimePartialResponseWriter (affects AJAX requests)
  • org.primefaces.context.PrimePartialViewContext (affects AJAX requests)
  • org.primefaces.application.PrimeResourceHandler (affects resource handling)

Head Resource Ordering Bug

PrimeFaces comes with some very cool features that allow you to control where your resources are going to be rendered in the <HEAD/>. You can use facets to place your resources into “first”, “middle” and “last” positions. This is a nice feature, since with RichFaces I had to place our style and script tags at the end of the <BODY/>, to make sure that they are rendered after the RF resources. This order is important, since it allows us to override RF styles and scripts.

However, there is a very obvious bug in org.primefaces.renderkit.HeadRenderer which makes this feature quite destructive to our app. Simply including the PrimeFaces jar causes all of our script and style overrides to stop working, b/c the app resources are placed before all others.

There are other issues with this feature as well: http://code.google.com/p/primefaces/issues/detail?id=4807

Workaround

The problem is explained here: http://code.google.com/p/primefaces/issues/detail?id=4807#c1. Even though I know how to fix the problem, I do not wish to start maintaining our own PrimeFaces patches at this point of low adoption. So, the workaround is to disable Head Resource Ordering entirely for now:

WEB-INF/faces-config.xml:

<renderer>
   <component-family>javax.faces.Output</component-family>
   <renderer-type>javax.faces.Head</renderer-type>
   <renderer-class>com.sun.faces.renderkit.html_basic.HeadRenderer</renderer-class>
</renderer>

This means that we now have to manually hardcode the theme include for PrimeFaces components:

WEB-INF/layout/mainTemplate.xhtml:

<h:outputStylesheet library="primefaces-bluesky" name="theme.css" target="head" />