Friday, April 26, 2013

Setting up Thrift and HDF5 on Windows 64-bit

(Note - this is also posted on the Crowbar Wiki  -Billy)
---

One of my projects uses the Thrift RPC library, *and* uses the HDF5 data storage file format, *and* must run on 64-bit Windows. Getting all these pieces to play nicely together was a huge pain, and this post documents what I went through to make it all work. I don't know if anyone else on the planet would need these all at once, but, well, here it is.

We're going to use Visual Studio 2012 Express, because it's free to use and seems to work. I spent an inordinate amount of time trying to use open source tools including Eclipse, MinGW, etc., and it took literally scores of hours and led nowhere. Don't do it. You're on Windows; use Visual Studio. Also, don't use VS Express 2010, because it doesn't have 64-bit compilers (I wasted a lot of time trying to hack around that, please don't waste your time like I did).

When this is all finished, you'll have the Visual Studio IDE set up, and all of the necessary libraries and includes in one neat folder under c:\dev.


0. Install Visual Studio 2012

If you have a license for Visual Studio 2012 Professional, great! Use that. Otherwise, use the no-cost Visual Studio 2012 Express for Windows Desktop. (not "For Web", not "For Windows 8").


1. Build Boost C++ Libraries

Boost is a highly-regarded set of C++ libraries that make all sorts of advanced programming tasks easier.  For this thrift project I need Boost Threads and Boost Shared Pointers, but there's no reason not to build the whole library. I'll probably need some other Boost stuff later, so, this builds the whole shebang:
  • Get Boost source (1.51 at this writing) from http://www.boost.org/ and unzip to your PC.
  • Build bjam, the boost builder.  Yes, Boost has its own builder:
    cd boost_1_51_0\tools\build\v2\engine;  build.bat  vc10
  • Build boost itself.  Some modules may not build under Windows, so I just leave those out.  The build command is a doozy, but if you type it correctly, it just works.

    CD to boost_1_51 and then:
tools\build\v2\engine\bin.ntx86\bjam.exe --prefix=c:\boost toolset=msvc address-model=64 variant=debug,release link=static,shared threading=multi --without-test --without-exception install
  • Advanced note:  after the above command built boost without the test and exception libraries, I was then able to build those libraries on their own using the same command, but changing --without-test --without-exception to --with-test --with-exception.  No idea why... but it worked.
  • Copy contents of c:\boost\lib into C:\dev\lib64, and move the entire c:\boost\include\boost-1.51\boost folder into C:\dev\include. Now Boost is built and can be used by all your VS projects. Yay!

2. Building HDF5 on Windows

The HDF5 source and related libraries are all available at http://www.hdfgroup.org/HDF5/release/obtain5.html — and first we need to build two dependencies for file compression: zlib and szip. The source links for zlib and szip are at the bottom of that page.
  • zlib: 
    • unpack, open the windows/static/zlib.vcproj file to import into VS. 
    • I had to remove and re-add all of the .c and .h files, but then it built fine.  Build all four debug/release/32/64 variants.
    • Copy the four .lib files from zlib-1.2.5/windows/static/dist into the /dev/lib32 and /dev/lib64 folders. 
    • Copy the .h files from zlib-1.2.5 *and* from zlib-1.2.5/windows into /dev/include
  • szip: same story.
Now we can build HDF5 itself:
  • Install CMake from www.cmake.org
  • Get the latest HDF5 source from http://www.hdfgroup.org/HDF5 and unzip it somewhere convenient (at time of writing, current source was hdf5-1.8.10-patch1, available here)
  • Create an empty "build folder" called "build"
  • Run the CMake gui, set the source and build folders. Click "Configure", choose Visual Studio 11 Win64.
  • After configure is finished the GUI will fill up with lots of options and checkboxes. For my project, pretty much everything is OK except I needed to check the "HDF5_Build_HL library". Add whatever else you need.
  • Don't muck with the ZLIB and SZIP options; we'll add those later.
  • Click "Add Entry" and add two options:
    • HDF5_NO_PACKAGES: (BOOL): ON
    • HDF5_USE_FOLDERS: (BOOL): OFF
  • Click "Generate".
  • You can now exit the CMake GUI.
  • Now we let cmake build the VS project files for us: all on one line, and be sure the "-Doption" lines don't have weird spaces.  Open a cmd window, cd into the BUILD folder, and:
    • cmake -G "Visual Studio 11 Win64" -DHDF5_ENABLE_SZIP_SUPPORT:BOOL=ON
      -DHDF5_ENABLE_Z_LIB_SUPPORT:BOOL=ON
      -DZLIB_INCLUDE_DIR:PATH=c:/dev/include
      -DZLIB_LIBRARY:FILEPATH=c:/dev/lib64/libzlib.lib
      -DSZIP_INCLUDE_DIR:PATH=c:/dev/include
      -DSZIP_LIBRARY:FILEPATH=c:/dev/lib64/libszip.lib
      -DHDF5_BUILD_FORTRAN:BOOL=OFF
      -DHDF5_BUILD_HL_LIB:BOOL=ON  
      ../hdf5-1.8.10-patch1
  • Now we can actually build; I built for both Debug and Release but maybe you just need one:
    • cmake --build . --config Release
  • Copy the libfiles from build/bin/Release into /dev/lib64; copy all src/*.h files into a new subfolder, /dev/include/hdf5.
  • Oh look!  Your toast is ready



3. Build Thrift


Thrift itself is actually dead-simple:
  • Get and unpack thrift 0.9.0 from thrift.apache.org
  • Open lib/cpp/libthrift.vcxproj in VS2010
  • Right-click the libthrift project / properties / VC++ Directories / add to the Include Directories: an entry for C:\dev\include
  • Build by right-clicking libthrift and choosing "Build".  
  • Repeat for each Debug/Release and 32/64 that you need.
  • Copy the built libs /dev/libXX . Note that for some reason this build doesn't add a "d" suffix to the debug versions; add it yourself or you will overwrite the release versions. Stupid.
  • This build also doesn't put the .h files anywhere nice for you; so I copy the entire lib/cpp/src/thrift folder into /dev/include/thrift, then I manually go thru every folder and remove the files that do not end in .h. 
  • Annoying, but you should be a pro at this by now.


4. Build log4cpp logging library


You may not need this one, but we needed a small decent logging library similar to log4j.
  • Get log4cpp and open the msvc10/msvc10.sln project solution. 
  • Right-clicky the log4cppLIB project and "Set as Startup Project"; 
  • Right-click and choose "Build" for debug/release and copy them to c:/dev.
  • Add/create the x64 target; then build Debug/Release again. It doesn't seem to put them in a separate x64 directory, so be sure you copied the win32 libs to c:/dev first.
  • Copy log4cpp/include/log4cpp to c:/dev/include

5. Get drunk

You now have a fully-populated c:\dev folder with all the includes and libraries needed to build and run Crowbar.  That took several hours, didn't it?  You deserve some tequila. Lots and lots of tequila.

No comments: