GATHER

How are we doing?

hladd — Fri, 15 Aug 2008 19:21:51 +0000



GATHER* is gathering momentum!  We are near completion
of the first field test of the platform in Uganda, and
while we have some small bugs to fix, we are feeling
good about how the system worked. The first use was
to expedite the regular disease surveillance and
reporting system in 20 health centers in Homia district. 

Ugandan health centers are required to report each week on
13 diseases and on other emerging infections suspected,
including polio, rabies, cholera, dysentery, measles, malaria
and other infectious diseases.  The current practices require
the completion of two paper forms that must be taken to the
district or sub-district office, often by foot or by bicycle. 
Reporting delays are common.  This approach is pretty common
in Africa with village and community health centers often in
remote areas without good transportation.

GATHER allowed the health workers to complete their reports
on cell phones and to send that data over the cellular network
to a server at the Ministry of Health. The GATHER server
confirms delivery, sends the data to the appropriate
district office and puts into the legacy database at the
Ministry of Health.   GATHER is designed to alert health
officials when reported numbers are outside of the norm
and may indicate an outbreak.

Recently implemented international health regulations require
UN member countries to report outbreaks on certain infectious
diseases within 24 hours.  GATHER has expedited reporting of
this critical data from rural health settings and has improved
the Ministry’s ability to comply with the regulations and to
quickly respond to outbreaks.

*GATHER is an opensource platform for data collection
and reporting, designed to work with multiple electronic
devices, developed by AED-SATELLIFE and Dimagi.  We are
gathering the best open source elements into the
platform.  Gather applications, gather data, gather
people, gather and share knowledge.  The current phase
of development is supported by a grant from the
Rockefeller Foundation.

Reporting – Pentaho Tools

gcapalbo — Mon, 30 Jun 2008 18:31:21 +0000

Browsing through Pentaho’s downloads section can be a bit intimidating. With such a wide array of products and tools, it’s a little hard at first to figure out the best way to start writing reports. So, I thought I’d post an entry on what I’ve found to be the most useful development environment for creating and editing reports.

Report Components

Pentaho reports follow a basic Model-View-Controller pattern. In it’s simplest form, a report consists of a .xaction file, which contains the report’s business logic, and a .xml file, which contains the report’s presentation logic. Optionally, .properties files can be used to include internationalized messages.

Main Tools

The two most important tools are the Report Designer and The Design Studio. Both can be downloaded from the Pentaho downloads section. I’ve been using the 1.7.0 RC2 versions of both, and both tools are very easy to install. In both cases, just unpack the zipped archive, and run the executable available (can be .exe, .bat, or .sh, depending on your operating system).

You’ll want to use the Report Designer to modify the view of your reports (.xml files), and the Design Studio for everything else. Before we get into the details of that, let’s first set up our environment.

Getting Setup

Once installed, the Report Designer requires no additional setup, but the Design Studio does. The Design Studio is basically an Eclipse development environment with a series of plugins geared towards fine tuning report files. To get setup, once it’s running, do the following:

Go to File->New Project
Select General->Project
Name the Project Pentaho Solutions
Uncheck the Use default location checkbox, and input the path to your pentaho-solutions directory.
Click Finish

Start Creating Reports

Now you can start reporting. I’ve found it most useful to always start with the Report Designer, because it is the easiest way to get a template up for your report even if you need to mold it later on with the Design Studio.

To start a new report, once the Report Designer is running, click on the Go button in the Step 1. Create New Report box. This will open up the Report Design Wizard. Follow the steps to create a basic outline for your report. All of this information can be changed later, so don’t worry if you need something more specific than what is provided.

TIP: Since you may want to fine tune your database query later on anyway (such as inject inputs from the user into your query syntax), you don’t even have to specify a complex of the query here. The most important part is specifying all of the columns you’d like to report on in your query so that it will automatically generate the columns to report on, so something like
SELECT 'a' AS COL1, 1 AS COL2 FROM TABLE
is fine for now. Later you can use the Design Studio to fine tune the query, as long as you make sure that the column aliases match up.

Once you are done with the Report Design Wizard, you can click the “Ok” button and you should see the outline of your report. If you do not see the outline of your report, you may have to click on the “Design” button at the bottom of the Report Designer preview window.

Now it’s a good idea to publish what you have so far. To do so:

1. Go to File->Publish
2. Input a path and name for the report’s .xml file (this should point to somewhere in your pentaho-solutions directory). All of the files for your report will have this name with a different extension (.xml, .xaction, .properties).
3. Check the Publish XAction File box. This will automatically populate the XAction File name.
Give the report a title and description.
4. Select xml in the Type drop down menu. This will allow the user to choose the output type of the report.
5. Click “OK”

Now you can open your .xaction file in the Design Studio and start editing it. You may have to first refresh your file tree in the Design Studio by right clicking on the root directory and clicking Refresh.

From here on out, it’s best to only use the Report Designer for modifying the .xml file. So, when you publish from the Report Designer after the initial publish, DO NOT check the Publish XAction File box, since this will overwrite any changes to your .xaction file.

Well, that does it for a basic workflow for creating and editing reports. Below is some useful information regarding the Report Designer and Design Studio.

Report Designer

The Report Designer is mainly a tool for designing the look and feel of your report. You can add new elements to your report by dragging them from the Palette on the left side and into your report. You can also reference variables passed from your .xaction as follows:
$(variableName)
where “variableName” is the name of your variable. For more information on the Report Designer, see http://wiki.pentaho.com/display/Reporting/Report+Designer.

Design Studio

The Design Studio lets you use a GUI to edit your .xaction file so you don’t have to do it by hand. A very useful component to the .xaction is the ability to embed JavaScript processing, which can be used to build a query or prepare inputs for display. The JavaScript can also reference Java classes by specifying them with a fully qualified name. For example,
var date = new java.util.Date();
will create a new Date object. You can also pass variables to your report view by clicking on the Define Process tab of your .xaction file, then clicking on the report Process Action and adding Report Parameters. For more information on the Design Studio, see http://wiki.pentaho.com/display/studio/Getting+Started+with+Design+Studio.

Summary

So, the basic workflow of creating a new report is as follows:

Use the Report Designer to start a basic outline of the report, and publish all of the report’s files (.xml and .xaction included) to your solution directory.
From then on,
- Use the Report Designer to only modify and publish the .xml file of the report.
- Use the Design Studio to modify all other files, most importantly, the .xaction file

-Giovanni

Developer instructions up on the wiki

jljackson — Thu, 05 Jun 2008 04:09:15 +0000

Andreas has put up a first pass at instructions for building the GATHER application completely from SVN and other 3rd party installations (will require Maven and Pentaho among others). If you are in the mood for an adventure, try them out and let us know how it goes. We haven’t had a chance yet to verify these on a clean developer machine.

This is a lot like a screencast, but all on the command-line and without any screenshots or voice-over.

http://code.dimagi.com/Gather/wiki/DeveloperGettingStarted

-Jon

Packaging the GATHER Appliance

Andreas Kollegger — Wed, 21 May 2008 18:47:23 +0000

Ease of use is on everyone’s short list of design goals for everything. Yet, project installation ranges from the elegance of Instiki’s “there is no step three” to the week-long exercise in mind-reading required for many “enterprise” applications. While we’d prefer to package GATHER into a simple executable jar file, having two separate web applications and a database makes that awkward.

Considering that our deployment scenarios always involve installing a physical server, we decided that our best option was to package GATHER into an appliance using the excellent tools available from rBuilder Online. With rBuilder, we’d have these features:

JeOS (just enough OS) using a Conary-based system
Web-based management for system updates
VMware image, Amazon EC2 AMI, or installable DVD from the same recipe

That last bullet is a killer, allowing us to create a release that is easily installed for testing (using a VMware image), for public access (running on Amazon EC2), or for deployment to a dedicated server.

I’ll walk through the steps of creating a new build of the GATHER appliance. To play along, you’ll need an account on rBuilder Online, and if you’re a GATHER team-member contact me to add your account to the project.

rPath tools

There are three command-line tools used for building and deploying an appliance image:

conary - the package-management tool used for system administration
- also note the Conary Commands wiki category
- this is a sysadmin tool for maintaining a conary-based system
cvc – the basic build tool, used mostly for source code management-like commands
- think of this as cvs or svn for conary recipe development
rmake - comprehensive build tool for developing appliances
- uses cvc for some operations
- always use an rmake command instead of the cvc equivalent

Get an Appliance Build Environment

Developing an appliance with rPath requires a conary-based system with the standard set of build tools. The easiest way to get that set up is to download and run the App2App image, which was custom built for the rPath Application to Appliance guide, but is perfect for general appliance development.

Download and install the free VMware Player (if you have Windows of Linux. Mac users will have to buy VMware Fusion or experiment with one of the open source virtualization options)
Download the App2App Image from rPath
Launch the App2App image
Log in using username/password = devuser/password

Now you are logged in to a conary-based system. If you look in the home directory you’ll notice that it is pre-populated with a development directory for an example project. For GATHER (or your own project), there are a few more steps.

Set up the developer Environment

Edit the .conaryrc file in the devuser home directory, editing the contact, name and user to match your user account on rbuilder, and adding the following section at the end of the file:

[gather-1-devel]
buildLabel gather.rpath.org@dimagi:gather-1-devel
installLabelPath gather.rpath.org@dimagi:gather-1-devel conary.rpath.com@rpl:1 ta.rpath.org@rpl:1 raa.rpath.org@rpath:rapa-3

That defines a context named “gather-1-devel” for working with package. The buildLabel indicates the branch of development. The installLabelPath indicates a search path for getting packages used by the branch.

Having defined a context for the branch, create a working directory and associate it with the context using ‘cvc context‘:

$ mkdir -p conary/gather/1-devel/
$ cd conary/gather/1-devel/
$ cvc context gather-1-devel

Modify any packages as needed

Now you can use cvc to check out any of the packages from the GATHER appliance project on rPath. The top-level package is “group-gather”, containing an appliance recipe which defines all the packages needed to build the image. You can checkout it, or any other GATHER package, with these steps:

$ cvc co group-gather
$ cd group-gather
$ ls
CONARY group-gather.recipe

If any of the packages have been changed, the group-gather package must be rebuilt and committed before a new image can be built using rBuilder online. The build will take a long time to complete as rmake fetches and assembles everything needed for a complete appliance, from the OS kernel to your packages. From within the group-gather directory, build the recipe using ‘rmake build‘:

$ rmake build group-gather.recipe

Note the job number, mentioned at the start of the build and appearing in square brackets throughout the build messages. In this example, the build is number 2. This is the number to use when committing the build. Commit the built binaries to the repository using ‘rmake commit‘:

$ rmake commit 2 -m "an example build of the group"

That can also take a long time to complete, as the binaries are uploaded to the online repository. Note the version reported at the end of the build.

Build a virtual machine image

Having committed a build that you’d like to distribute, switch to a web browser to use rBuilder Online.

Start at the GATHER project page
Click on “Manage Builds” in the left column
Click “Create a new build“
Select the kind of virtual machine you’d like to make under “Build Types”
Under “Build Contents” click through the following sequence
1. group-gather – the appliance package recipe
2. gather.rpath.org@dimagi:gather-1-devel – the branch to build
3. the version of group-gather you had committed – something like 0.1-17-1
4. the only flavor available
5. now the “Create Build” button should be active
Click “Create Build”

rBuilder Online will queue up the build request and present a page where you can monitor the progress. It’s ok to leave this page, the job will continue until it succeeds or fails. When it is done, the new image should be available on the build page of the project, under “Individual Builds”. These are only available to project members. A release must be built using the Manage Releases page to publish the build to the general community.

Simple, right?

-Andreas

Reporting – Getting Started with Pentaho

Andreas Kollegger — Tue, 20 May 2008 21:41:49 +0000

In the three-step waltz of gather, transform, report, it’s that third step that gets the accent. Reporting provides the real value of a form capture system. GATHER is concerned with the larger themes and chords that can orchestrate nationwide public health efforts.

So, selecting a reporting framework was our most important decision. After some review (which I’ll get into later) we chose the Pentaho open source business intelligence suite.

Within GATHER, Pentaho exists as a separate web application, integrated by a common data source, some branding, and custom reports. Before working with the GATHER version, it is best to get started with the standalone Pentaho application to really understand what it offers and how it can be used.

Here are the steps I recommend for getting started with Pentaho:

Start-up: download, install and run Pentaho
Try it out: browse around through the web application
Write a report: use the Pentaho Report Design Wizard to write a report
Learn more: refer to the official Pentaho documents

Start-up

Download Pentaho Open BI Suite, Pre-configured Installation (PCI) – Hypersonic
- this is the full stand-alone Pentaho web application, complete with an application container (JBoss)
- grab version 1.7.0.RC2 for your platform
- Hypersonic means using HSQLDB, which is much simpler for development and testing
- yes, the download is giant
Download Pentaho Report Design Wizard Software
- this is the easiest way to get started writing reports
- grab version 1.7.0.RC2 for your platform
Start up the server
- use the command-line script for your platform (example: “start-pentaho.sh” for Mac OS X)
- Troubleshooting
  - port 8080 should be available (no Tomcat or other container should be running)
  - JBOSS_HOME environment variable should not be set

Despite many warnings and possibly some errors, a message including the phrase “Pentaho BI Platform server is ready” indicates that the Hypersonic database and JBOSS have both launched, and Pentaho is ready.

Try it out

Go to http://localhost:8080/pentaho/
Select a user from the “Valid Users” drop-down, then hit the “Login” button
From the “Go” menu, select “Solutions”
- for GATHER, the “Samples” solution is replaced by a “GATHER” solution
Browse through the examples

The main areas of interest are the examples in the Solution Browser, and the ad-hoc report generator available under “Go -> New Report”. For GATHER, adding to the pre-written reports is important.

Write a report

The easiest way to get started writing reports for Pentaho is to use the Report Design Wizard . Follow through the step-by-step guide to learn how to use the wizard. The result will be a set of files defining the report and an xaction for integrating the report into Pentaho. This collection must be published into a Pentaho solution. You can try publishing directly to the server, or by following these manual steps:

In the final step of the “Pentaho Report Design Wizard”, click the “Publish” button
Choose “location”
Browse to the “pentaho-demo/pentaho-solutions/samples” directory. Create a new directory for your report.
Uncheck “Create JBoss Datasource”
Click the “OK” button

To make the new report visible to the Pentaho Solutions browser, the directory must contain an index.xml file and an index.properties file. Just copy these (and possibly an icon file like file.png) from an existing sample report. Edit the index files to reflect your preferences. Then, follow these steps in the Pentaho web application:

Click the “Admin” menu
Click the “Update Solution Repository” action
- click OK in the confirmation dialog
- click Close in the “Publisher Administration” dialog which confirms that the update is complete
Click the “Go -> Solutions” menu item
Look for your report in the list (the name will be the name defined in your index.properties file)

Learn more

Creating Pentaho Solutions - a “solution” is an organizational unit for Pentaho, a workflow containing actions like running a report or emailing the results of a report
Styling Pentaho – an under-documented how-to for customizing Pentaho. Artifacts which contribute to the style include:
- pentaho.war – the actual jsp pages of the web application
- pentaho-style.war – stylesheets and images
- pentaho-solutions – images and text used by the solution browser
Pentaho Design Studio – an Eclipse-based suite of editors for creating action documents and reports

This is a very brief introduction to Pentaho Reporting. There is a lot more to explore and leverage for GATHER users, which we’ll cover in future posts.

Cheers,

Andreas

From the Field

Andreas Kollegger — Tue, 20 May 2008 02:52:19 +0000

Early in the design process, we acknowledged that the major functional components of GATHER existed in other open source projects. Rather than grow our own, we would focus on bringing together the best solutions available in the field — integration over invention.

The impulse to grow your own, or reinvent the wheel, isn’t just about hubris, however. Sometimes it can just be easier and quicker to make your own rounded spinning thingy than to figure out how to propel someone else’s. Balancing adoption & adaption against original development was and remains our biggest challenge, compounding the usual difficulty of estimating software schedules.

We started with a field survey. Here’s brief summary of what we considered:

Build Tools

Ant - the ubiquitous java build tool.
Ivy – dependency management. Works in conjunction with ant.
Maven - build management with an object-oriented project definition. Handles dependencies, builds, test, deployment.
Cargo – a thin wrapper for manipulating application containers. Include Maven plugin.
Subversion – popular source code management (SCM) solution
Mercurial – lightweight SCM for decentralized, distributed management

Application Framework

Spring Framework – web application framework which promotes modular application assembly and provides commonly required functionality
OpenESB – open source ESB from java.net
Mule – an open source enterprise service bus (ESB), simplifying the process of gluing services together
ServiceMix – Apache’s JBI compliant ESB

Application Features

XML
- XForms – next generation forms controls for html
- Castor – XML data binding product. XML to POJO, POJO to XML, POJO to relational database.
- Formfaces – client-side cross-browser javascript library for adding XForms support
- Hyperjaxb2 – JAXB implementation with persistence
- JAXB – reference implementation of JAXB (java api for XML binding)
- JaxMe2 – apache open source implementation of JAXB, with persistence
- Orbeon – web application which presents Xforms using standard html
Persistence
- Java Persistence API (JPA) – official java library for storing objects in or mapping objects to a relational database
- Hibernate- object/relational persistence and query service (JPA compatible)
- iBATIS- object/relational mapping using XML descriptors
- MySQL- “the world’s most popular open source database”
- HSQLDB- fast, reliable, embeddable java-based DB
- Derby- Apache’s java-based DB
- H2- very capable java SQL DB, successor to HSQLDB
Reporting
- Pentaho – suite of data reporting, analysis, transform and mining tools
- Jasper Reports – suite of data reporting, analysis, transform and mining tools
- Pentaho – suite of data reporting, analysis, transform and mining tools
Scripting
- Groovy – an agile dynamic language for Java
- BSF – Bean Scripting Framework, supporting several languages
User Management
- Acegi – security solution for enterprise applications

Packaging & Deployment

Getdown – a java application install/update framework (JNLP style)
Web Start – Sun’s framework for install/update of applications from a web page
jsvc – java daemon service, for managed headless java applications on unix
Procrun – tool for running java applications as a service on Win32
Tomcat – the robust Apache Foundation servlet container
Jetty – a stand-alone or embeddable web server and servlet container

So, what did we actually end up using? We’ll cover that in the next couple of posts.

-Andreas

SMS Projects

jljackson — Wed, 14 May 2008 20:59:45 +0000

SMS will become a supported interface by GATHER in the near future, and we have been speaking with a variety of groups about their SMS projects. We will be integrating the best of breed tools from a variety of different SMS projects, but wanted to highlight two projects that we just looked at:

1. A small project some of the Dimagi develolpers created on can be found here: http://code.dimagi.com/SmsGdata. It demonstrates the ability to use Google spreadsheets to design a lightweight dataset to be collected and receives an SMS from phones in India. This was deployed in May 2008.

2. We just had a great discussion with a technical group at Unicef that we are excited to follow-up with. They have several interest uses of SMS and a toolkit that looks incredibly powerful http://x.mepemepe.com. We will hopefully be able to meet with their technical team sometime in May to discuss collaborative oppurtunities.

Possible Logos

Andreas Kollegger — Thu, 24 Apr 2008 21:28:42 +0000

Every project more involved than spring cleaning your house deserve a good logo. Here are two options with which I’ve been playing around, the gather(ing) basket, and gather(ed) honeycomb:

Also, given my impulse to use parentheses, here is the possible (product) red edition of the honeycomb:

Reactions?

-Andreas

gather, transform, report

Andreas Kollegger — Tue, 22 Apr 2008 21:09:56 +0000

Conducting a field study is simple: gather data from the field, transform the data into an analyzable format, then produce reports on the data set. Whether conducting environmental impact studies, researching the migration of an endangered species, or monitoring public health, the pattern is consistent: gather, transform, report.

When I had first started working in public health, for SmartCare in Zambia, that pattern was one of many that frequently came up in discussion among the developers. A research project has lots of options for implementing the steps – paper-based, simple database, or enterprise systems.

Jonathan Jackson observed that scaling from paper forms, to a Microsoft Access solution, then on to a custom enterprise solution is always painful. We talked about designing a domain-neutral system that could scale gracefully from a single point solution to a distributed system.

GATHER is our modular framework for providing a scalable solution for data capture and reporting, an integration of the best available open source solutions. The initial implementation will feature:

Gather – data capture by mobile clients from the OpenRosa consortium
Transform – XForm instance data saved to relational tables
Report – integrate Pentaho for pre-designed and ad-hoc reporting

The initial implementation and funding for this effort is through AED-SATELLIFE, the Rockefeller Foundation, and Dimagi. We’re pretty excited to get everything up and running. When we have a software distribution ready, we’ll make announcements here and on the gather mailing list.

Cheers,

Andreas