Work done

This last week, I fixed the last found errors in the existing code. I planned to implement more pending features, but after talking with my mentors, I decided to review, fix and test the existing code. So, after manually fixing some easy-to-find bugs, I exported the GDAL autotest code from https://svn.osgeo.org/gdal/trunk/autotest and created my own test script, using python.

With this script, I successfully tested:

• Get GDAL driver by name
• Try to open a dataset using wrong connection strings
• Compare, pixel by pixel, 2 WKT Rasters (1 band and 3 bands) with the original files
• Open 2 WKT Rasters (1 band and 3 bands) and compare projection references and geotransform arrays with the original files

There are 9 tests. The necessary data for the test have been added to the gdrivers/data directory. You’ll need:

• A loading script, to execute gdal2wktraster loader and psql to add the new data to PostGIS
• Example of image with one band
• Example of image with three bands

I successfully used GDAL tools like gdalinfo, gdal_translate or gdalwarp to test the code too. And I tried to use gdal2tiles, but the code needs a few modifications to accept a connection string as a valid dataset.  Oh, and I improved my version of gdal2wktraster script to add outdb support. I’ve updated the patch in the official osgeo ticket. So, you can download and test this new version. It still needs improvements, I’m sure.

There are two features that I couldn’t test, for different reasons, and this will be two of my first tasks after GSoC. These features are:

• Outdb support: After modifying and testing the gdal2wktraster script to add outdb support, I realized on one bug in my code: I was trying to read the outdb raster data from IReadBlock method. But the outdb raster doesn’t necessarily have the same block arrangement than the indb raster. So, the block offsets given in IReadBlock couldn’t be valid for the outdb raster. And  normally, they aren’t. I realized on this error too late, and I prefered to add a comment and continue developing this feature after GSoC. In addition, you have to hack the WKT Raster code to allow outdb raster support at WKT Raster extension’s level, because this feature is under development. I chose the short path: comment the outdb checking, in rt_pg/rtpostgis.sql file, lines 532 – 535.
• Endianess: I created the code for adding raster data to a testing table with a different endianess that my machine (little endian machine). But, when I tried to execute the “INSERT” instruction, the data was automatically changed to the correct endianess by the WKT Raster extension. I tried to hack the code, but I failed this time…

So, I have some untested features finished or partially finished (in the case of outdb support). And I finished other features, like setting raster data, or setting projection, geotransform array and nodata values. These features will be used for the creation of new WKT Rasters from GDAL driver.

Anyway, if you want to see the complete list of TODO tasks (always under revision), check the project page. I think the driver, in its current state, is a stable prototype GDAL driver, and it will be a complete one by the release of the version 1.7.0 of GDAL.

Here, you can check out the GDAL code, with the WKT Raster protoype code

Here, you can check out the GDAL autotest code, that includes the testing code for GDAL WKT Raster.

GSoC 09 conclussions

I think the GSoC 09 has been one of the greater experiences of my student’s life. For first time, I fell I’ve developed a really useful tool, in the frame of a excelent software like GDAL, an essential piece of the Open Source GIS-related world. I was a GDAL user, and now, with the permission of the GDAL team, I feel like a newbie GDAL developer. An incredible feeling

During this summer of code, I’ve acquired more programming skills than in the last year. I’ve learned:

• How to do some things in a smarter way, without “reinventing the wheel”.
• How to organize a big software project using C/C++
• How to write good makefiles
• How to implement a driver, meaning how to take advantage of basic OO features like inheritance, information hiding, polymorphism, data abstraction, modularity, encapsulation. The GDAL driver architecture is an excelent tool to see these concepts working.
• How to make questions. When you post a message to a developer list, it’s important that you provide a thorough description of the error, an example output, the version of the libraries you’re using, the configuration of your working enviroment… Obvious? Of course, but you need to be in the situation to really realize on how to ask for help.
• Related with previous point, how to feel totally stupid when you find the error just after cliking on the “Send” button. An unique feeling
• How to organize myself. Yes, it may sound stupid, but you don’t know how organized are you  until you have a chance like this.
• How to sleep less than 4 hours and write (working) code . This drives us to an eternal discussion: tea or coffee? Tea works better for me…

The future…

In the future, I plan to finish the driver code, of course. Now, I’m personally commited with it. I should finish the driver’s code for the release of GDAL v1.7.0, and I’m going to work in this way. As I said, you can check the todo tasks, and the planned schedule (always under revision), in the project page.

Last, but not least, I’d like to thank my mentors, and all the GDAL and PostGIS development teams members for their patience, solving all my doubts. Of course, thanks to the OSGeo for giving me this chance of learning and helping the Open Source community.

Oh, and this is for Wolf Bergenheim: I think I escaped punishment of the Knights of Ni!, the keepers of the sacred words: Ni, Peng, and Neee-Wom (“Those who hear them seldom live to tell the tale!”). Will I have my T-Shirt?

Work done

This week:

• I’ve fixed a bug with overviews’ dimensions.
• I’ve successfully tested the overviews with dumpoverviews utility. I had to introduce small changes in the code to accept connections to PostgreSQL as input datasets. I created a patch for the original code, but it has no sense to use it without the rest of WKT Raster driver.
• I’ve added support for outdb rasters. It has to be tested.
• I’ve changed the way the driver gets the data from database. To minimize the rounds to the server, I’ve created a kind of “cache” system at Dataset’s level. After opening the dataset (establish a database connection, basically), The driver tries to fetch all the blocks covered by the whole raster’s extent. In case of success, it store all the blocks as WKTRasterWrapper objects in an array at Dataset. Then,WKTRasterRasterBand::IReadBlock method get the blocks from this array. If fails, execute a query to fetch it from database. I’m testing this cache system just now.

Planned work for next week

Next week is the last one. I’ll finish testing the tasks of this week and I’ll write final documentation and some testing scripts.

There are todo tasks, of course. But I’ve planned to continue with the project after GSoC, and my mentors think it’s a good idea.

Problems found

The way to implement the cache system. An option was to use the cache at RasterBand’s level (GDALBlocks), but one of my mentors suggested me other idea, and I adopted it.

Work done

This week I was out for 3 days due to an unexpected personal issue. So, I didn’t finish all my planned tasks. Now, I’m making up these days. Anyway, the work done was:

• Start the RASTER_COLUMNS table update with the information fetched from database.
• Create a patch for gdal2wktraster script and send it to the PostGIS track as new ticket, instead of sending the modified script to the list. I’ve delayed the out-db raster support in GDAL because is under development in WKTRaster, and I tried to help in this task.
• Correct errors. I’ve introduced some changes, like erase the overviews from SUBDATASETS metadata domain, fix some memory leaks due to sharing objects between Datasets and overviews, add NBITS metadata information related with each RasterBand, and some other little fixes.
• Test overviews support with gdal_translate.
• Start block caching

Thanks to people from gdal and postgis lists (like Frank, Tamas, Mateusz, Even…) I solved the two doubts I had the last week:

DOUBT: What happens if the bb of a block offset in IReadBlock doesn’t match any real block? Missing-tiles raster?
RESPONSE: In this case, the buffer must be filled with nodata values

DOUBT: Where should I fit WKTRaster “16BF” datatype? In GDT_Uint16?, in a single float?
RESPONSE: It fits in a 32bits float. Anyway, this datatype is proposed to be deleted from WKTRaster: http://trac.osgeo.org/postgis/ticket/226

UPDATE: I forgot to add the graphical proof of overviews running:

Here, the file utm.tif, used for testing (reduced to fit into the blog’ space). Look at the dimensions, at bottom left corner.

Here, the result after loading the image in PostgreSQL with gdal2wktraster, using a block size of 100×100px and converting the image to TIFF format again using gdal_translate with an output size of 50%

Planned work for next week

This week must be the week to end the tasks related with regular_blocking support, and to start with non-regular blocking arrangements.

Problems found

One important problem was to share a PGconn object between one dataset and its overviews (datasets too). The connection is released in the class destructor, but only the general dataset should do it, as the connection owner. Finally, I solved the situation thanks to suggestions from Even Rouault and Mateusz Loskot, by using a boolean var to detect if it’s the dataset or one of its overviews.