Setting up Thrift and HDF5 on Windows 64-bit

(Note - this is also posted on the Crowbar Wiki  -Billy)
---

One of my projects uses the Thrift RPC library, *and* uses the HDF5 data storage file format, *and* must run on 64-bit Windows. Getting all these pieces to play nicely together was a huge pain, and this post documents what I went through to make it all work. I don't know if anyone else on the planet would need these all at once, but, well, here it is.

We're going to use Visual Studio 2012 Express, because it's free to use and seems to work. I spent an inordinate amount of time trying to use open source tools including Eclipse, MinGW, etc., and it took literally scores of hours and led nowhere. Don't do it. You're on Windows; use Visual Studio. Also, don't use VS Express 2010, because it doesn't have 64-bit compilers (I wasted a lot of time trying to hack around that, please don't waste your time like I did).

When this is all finished, you'll have the Visual Studio IDE set up, and all of the necessary libraries and includes in one neat folder under c:\dev.

0. Install Visual Studio 2012

If you have a license for Visual Studio 2012 Professional, great! Use that. Otherwise, use the no-cost Visual Studio 2012 Express for Windows Desktop. (not "For Web", not "For Windows 8").

1. Build Boost C++ Libraries

Boost is a highly-regarded set of C++ libraries that make all sorts of advanced programming tasks easier.  For this thrift project I need Boost Threads and Boost Shared Pointers, but there's no reason not to build the whole library. I'll probably need some other Boost stuff later, so, this builds the whole shebang:

• Get Boost source (1.51 at this writing) from http://www.boost.org/ and unzip to your PC.
• Build bjam, the boost builder.  Yes, Boost has its own builder:
  cd boost_1_51_0\tools\build\v2\engine;  build.bat  vc10
• Build boost itself.  Some modules may not build under Windows, so I just leave those out.  The build command is a doozy, but if you type it correctly, it just works.
  
  CD to boost_1_51 and then:

tools\build\v2\engine\bin.ntx86\bjam.exe --prefix=c:\boost toolset=msvc address-model=64 variant=debug,release link=static,shared threading=multi --without-test --without-exception install

• Advanced note:  after the above command built boost without the test and exception libraries, I was then able to build those libraries on their own using the same command, but changing --without-test --without-exception to --with-test --with-exception.  No idea why... but it worked.
• Copy contents of c:\boost\lib into C:\dev\lib64, and move the entire c:\boost\include\boost-1.51\boost folder into C:\dev\include. Now Boost is built and can be used by all your VS projects. Yay!

2. Building HDF5 on Windows

The HDF5 source and related libraries are all available at http://www.hdfgroup.org/HDF5/release/obtain5.html — and first we need to build two dependencies for file compression: zlib and szip. The source links for zlib and szip are at the bottom of that page.

• zlib: 
• unpack, open the windows/static/zlib.vcproj file to import into VS. 
• I had to remove and re-add all of the .c and .h files, but then it built fine.  Build all four debug/release/32/64 variants.
• Copy the four .lib files from zlib-1.2.5/windows/static/dist into the /dev/lib32 and /dev/lib64 folders. 
• Copy the .h files from zlib-1.2.5 *and* from zlib-1.2.5/windows into /dev/include
• szip: same story.

Now we can build HDF5 itself:

• Install CMake from www.cmake.org
• Get the latest HDF5 source from http://www.hdfgroup.org/HDF5 and unzip it somewhere convenient (at time of writing, current source was hdf5-1.8.10-patch1, available here)
• Create an empty "build folder" called "build"
• Run the CMake gui, set the source and build folders. Click "Configure", choose Visual Studio 11 Win64.
• After configure is finished the GUI will fill up with lots of options and checkboxes. For my project, pretty much everything is OK except I needed to check the "HDF5_Build_HL library". Add whatever else you need.
• Don't muck with the ZLIB and SZIP options; we'll add those later.
• Click "Add Entry" and add two options:
• HDF5_NO_PACKAGES: (BOOL): ON
• HDF5_USE_FOLDERS: (BOOL): OFF
• Click "Generate".
• You can now exit the CMake GUI.
• Now we let cmake build the VS project files for us: all on one line, and be sure the "-Doption" lines don't have weird spaces.  Open a cmd window, cd into the BUILD folder, and:
• cmake -G "Visual Studio 11 Win64" -DHDF5_ENABLE_SZIP_SUPPORT:BOOL=ON
  -DHDF5_ENABLE_Z_LIB_SUPPORT:BOOL=ON
  -DZLIB_INCLUDE_DIR:PATH=c:/dev/include
  -DZLIB_LIBRARY:FILEPATH=c:/dev/lib64/libzlib.lib
  -DSZIP_INCLUDE_DIR:PATH=c:/dev/include
  -DSZIP_LIBRARY:FILEPATH=c:/dev/lib64/libszip.lib
  -DHDF5_BUILD_FORTRAN:BOOL=OFF
  -DHDF5_BUILD_HL_LIB:BOOL=ON  
  ../hdf5-1.8.10-patch1
• Now we can actually build; I built for both Debug and Release but maybe you just need one:
• cmake --build . --config Release
• Copy the libfiles from build/bin/Release into /dev/lib64; copy all src/*.h files into a new subfolder, /dev/include/hdf5.
• Oh look!  Your toast is ready. 

3. Build Thrift

Thrift itself is actually dead-simple:

• Get and unpack thrift 0.9.0 from thrift.apache.org
• Open lib/cpp/libthrift.vcxproj in VS2010
• Right-click the libthrift project / properties / VC++ Directories / add to the Include Directories: an entry for C:\dev\include
• Build by right-clicking libthrift and choosing "Build".  
• Repeat for each Debug/Release and 32/64 that you need.
• Copy the built libs /dev/libXX . Note that for some reason this build doesn't add a "d" suffix to the debug versions; add it yourself or you will overwrite the release versions. Stupid.
• This build also doesn't put the .h files anywhere nice for you; so I copy the entire lib/cpp/src/thrift folder into /dev/include/thrift, then I manually go thru every folder and remove the files that do not end in .h. 
• Annoying, but you should be a pro at this by now.

4. Build log4cpp logging library

You may not need this one, but we needed a small decent logging library similar to log4j.

• Get log4cpp and open the msvc10/msvc10.sln project solution. 
• Right-clicky the log4cppLIB project and "Set as Startup Project"; 
• Right-click and choose "Build" for debug/release and copy them to c:/dev.
• Add/create the x64 target; then build Debug/Release again. It doesn't seem to put them in a separate x64 directory, so be sure you copied the win32 libs to c:/dev first.
• Copy log4cpp/include/log4cpp to c:/dev/include

5. Get drunk

You now have a fully-populated c:\dev folder with all the includes and libraries needed to build and run Crowbar.  That took several hours, didn't it?  You deserve some tequila. Lots and lots of tequila.

12 Months

One year ago today, I quit my job.  Some highlights and rumination on the past year, at...
https://plus.google.com/photos/110126738379882279569/albums/5760364065080248113

IT. WORKED.

Stop eating sugar.  Now.

I just returned from my doctor's office, half a year after blood tests revealed I had high triglycerides, low testosterone, and medium (but not awful) cholesterol levels.  I also had "fatty intrusions" in my liver, which can lead to metabolic syndrome and diabetes. My doctor said to come back in six months and see if the problems persisted, and if they did, we would have to consider prescriptions or worse.  But it turns out, all of those problems were fixable.

Sparky and I decided we were getting way too out of shape anyway, and that we should try to fix things naturally first. As a joke we decided to target "Bear Week" on Cape Cod as our target date to be in better shape.  That gave us about four months.

I thought we would just be eating less and exercising more.  Yes, we did those things with varying success. But two things happened at about the same time:  first we stumbled across the New York Times Magazine article Is Sugar Toxic? which goes into great detail about how some nutritionists now think eating sugar, not eating fat, may be the real culprit in causing weight gain, diabetes, heart attacks – even some types of cancer.  Read it, it's worth the time.

Then, about a week later on a Saturday afternoon, we stumbled into our friend Paul at the El Toro taqueria in the Mission.  He had clearly lost at least twenty pounds, so we picked his brain while we ate together. At first it seemed strange watching him eat at a Mexican taqueria — surely horrible for a diet! — but he left out the rice, didn't eat the tortilla, and doubled up on the meat, the black beans, and the veggies. The secret? Cut out all the sugar, and most of the white carbs from his diet.  He pointed us to a Tim Ferriss book which we never bought and never read, but poking around the web we were able to put together most of what it suggested anyway.

For several months, six days a week, we cut out as much sugar and white carbs as humanly possible from our diets.  No cereal, no chinese food dripping in candy "spicy" sauce, no noodles, no rice, no bread.  Beans are OK since they're high in protein.  Meat, veggies, fine.  No fruit!  No sodas.  Not even diet sodas. Then one day a week, we had "off day" and ate whatever we wanted, from ice-cream and pineapple waffles to pork fried rice and back again.  By binging once a week, we never felt like we were suffering or skipping things we wanted.  I actually kept a list all week of things I wanted to eat on Saturday.  It was brilliant.

So the end result? As of today, my cholesterol ratio (<5.0 is good) went down from 4.5 to 3.7; my triglycerides (<150 is good) down from 183 to 146, and my testosterone (>350 good) up from the high 200s to 375.  All of them are now in their normal ranges.  I also lost fifteen pounds, despite stuffing myself with steak and eggs for six months.

I know this is only one data point, and everyone is different. But I am now convinced that the food industry has been shoveling low-fat propaganda down our throats for so many decades that we can't even think straight about what's healthy to eat and what isn't.  It's not the fat that makes you fat.  It's the sugar.

All my Macs

So this isn't about Steve Jobs, but it is about Apple. The first computer I paid for was a Mac IIsi in 1991, and it was one of the very first things I bought when I got a job that summer after graduating from college. (Note that back then I was only recently-converted to the Apple universe; I was a die-hard Atari computer fan during the 8-bit wars of the 1980's). I got a color monitor and a 2400 baud modem, and immediately started downloading... ahem well anyway some things haven't changed.

Through the '90s I got my parents a Mac Performa, and later a ruby red iMac, and all the while ridiculed Microsoft well beyond the intro of Windows 95 and its revolutionary long filenames. That first Mac got me onto the fledgling internet, and I learned FTP and gopher from the W.E.L.L. during one very droughted winter in San Francisco. I did end up getting a Win95 PC in graduate school so I could have a cheap copy of SPSS for my statistics-heavy thesis, but even then I was trying out alternative OS'es including Linux and even BeOS; I never was really that into Windows.

But something changed in the early 2000's. My ex-husband and I both agreed that the world would be even more draconian and horrible if Apple had actually beaten Microsoft in the '90s: more vendor lock-in, fewer choices, etc. I bought my last Mac in 2005, one of the original Mac Minis. It was a horrible, slow machine, and I sold it to a friend within months. Such a disappointment!! I then built my own PC and tried every flavor of Linux possible, all while trying to resist using Windows whenever I didn't need to. Installing and re-installing countless versions of Ubuntu, Fedora, and MS operating systems became a bit of a joke between me and my ex; it seemed every Saturday I had to do hours of computer maintenance just to get my video or sound working. Meanwhile he just played Everquest all day, waiting for me to be ready to play. I think eventually he gave up.

I still haven't forgiven Apple for my Mac Mini disaster. Meanwhile I skipped Vista and have been very functional with Windows 7, although it's a low bar to be impressed just because it doesn't crash and the sound & printer drivers all seem to work. Maybe Windows 8 will be a revolution; I hope so.

As for the vendor lock-in, I think everyone who's joined the Apple ecosystem knows what I'm talking about. It's a great system as long as you stay inside the garden and only use Apple products: their computers, tablets, phones, and even streaming TV boxes all work in glorious harmony together. Just don't ever ask for a black laptop, because they only come in one color.

Now I've got a Windows 7 PC, an Ubuntu laptop, a Chromebook, and an Android phone. I hope Apple rediscovers innovating instead of litigating. They keep the whole tech sector chasing taillights when they try their hardest, and I'd love to feel that magic again someday.

New online security measures

[edited to make clearer and shorter!]

Sparky had his Gmail account broken into by Chinese hackers last week, most likely because of the Sony Playstation security breach earlier this summer. He's going to be okay, but we both decided it was time to get serious about protecting ourselves and I wrote it all down in case anyone out there is thinking of doing something similar.

The first few steps take less than 30 minutes to set up. Do them! The last step takes a few hours but really that's just one Saturday morning, and I think it's well worth the time. These steps will make it more difficult for hackers to break into your computer, and will limit the damage if an online site you already trust (like your bank, or eBay, or Sony) gets hacked and your login info at that "trusted" site gets stolen.

---

The short version: I only have to remember three really secure passwords.

1. The first is for my main computer/laptop, which is set to require a password on start-up, on wake-up from sleep, and even from the screensaver, which kicks in after just a few minutes.
2. The second is for my Gmail/Google account, which is important because any website I go to has a "Forgot password?" link which would send that site's password to my Gmail! So that's critical.
3. The third is a master password which protects a file listing all my other website passwords, easily accessible from my desktop and my phone.

Now, the details:

---

1. Set your computers and laptops to lock with a password when they sleep or shut down. (takes 5 minutes)

If someone steals your laptop, it's as if a burglar broke into your home and stole your filing cabinet. It's so easy to protect yourself, and yet these settings aren't the default:

• Use a strong password for your account login.
  
  
• Set your machine to require that password on "wake up", so you must know the password to get into your account. On Windows and Mac this is under the Power Settings in the control panel. This protects you from someone opening your web browser and jumping straight to your mail or other websites which probably "remember you".
  
  
• Set up a Guest Account on your computer so friends and visitors can use it easily without having to type your password. It's so simple: on both Mac and Windows it's just in the User Accounts control panel.

• Set a lock timer so that the screen automatically locks after 5 minutes. This is also in your control panel, either under "Screen Saver" or "Power Settings".

In many ways this is the most important step, because if a thief walks off with your laptop and it doesn't lock, your browser probably remembers all of your logins and won't even bother to ask for a password when the intruder goes to gmail.com or facebook, or whatever. So having the machine lock is a small bother, but a huge security win.

(If you're on a Mac, you can optionally turn on "hard drive encryption" for your user account. This makes it even harder for a thief to get your information, so long as he doesn't know your account password. On Windows this is not a built-in feature, but you can use third-party software such as TrueCrypt if you want to do this. I have not set up full-disk encryption and probably won't.)

---

2. If you're using Gmail as your primary email, turn on "two-factor" security. (15 minutes)

Two-factor is the online equivalent of an ATM card: you need your password AND a physical device (usually your cellphone) to log in from a new location or PC. The login attempt causes Gmail to send you an SMS message to your phone with a one-time PIN number that you then key in. Without the PIN, you can't login.

With two-factor, the Chinese hackers would not have gotten into Sparky's mail, even if they had stolen his password, because they could never have received the extra PIN number unless they had also physically stolen his phone. See this two-factor explanation from Google for more info.

Set it up carefully, the details are important! If you have a smartphone, you can also use an app to get the PIN instead of waiting for an SMS. It's pretty cool.

---

3a. Stop using the same passwords for all your websites and accounts. Change ALL of them.

If you're like me and just about everyone else out there, you probably have a small handful of passwords memorized, and you use the same ones everywhere. Maybe your banking password is harder to remember than the one you use for Facebook or that book-sharing site. This is a terrible idea, because if one of those sites gets hacked, they have the keys to all of your online identities. You KNOW this is a bad idea, but you don't want to memorize 30 passwords.

Especially for sites that really are the key to your identity, such as your primary email, it's absolutely critical that you use a separate login with a really tough password. Someone with access to my browser and my Gmail, for example, could go to every site in my history list, click "I forgot my password", and have my password for that site emailed to them.

Scary, huh... and the solution?

---

3b. Use a password manager to create and save all your new, tough passwords (a few hours)

I installed KeyPass on my desktop computer, and on my Android phone. It's just a password manager; it requires you type in a master password before it shows all your stored passwords. It's free and open source, which means it's very secure, and there are versions for Windows, Mac, Android, Blackberry, and iPhone. It lets you group and sort passwords any way you want; and it can auto-generate long, random passwords for every site you use. There are other similar apps too (this isn't an advertisement!).

Make sure your master password is very strong and very memorable; if you lose it, you're screwed. See this xkcd comic for a reminder of what good and bad passwords look like!

I trust the security in my main browser to save my passwords, so as long as my computers are safe and locked down (see below), I don't have to type any of these long passwords in more than once in a while. And KeePass lets you cut/paste the passwords anyway, so actually there's no typing at all. KeePass is great.

So I logged into every website I use, navigated to the "Manage Account" page, and changed my password to a random one suggested by KeePass. Some websites enforce length, numbers, letters, spaces, no spaces, etc -- just check the right boxes in KeePass and it will do it for you. Then, after all of my logins were reset, I copied the KeePass database file from my PC to my Android phone. Now I have all those complex passwords anywhere I need them.

---

4. Keep your security software up-to-date.

Boring but important: make sure that your software updates are set to run automatically -- both system updates, and antivirus/malware such as MS Security Essentials. And if you're on Windows, I gotta say... just uninstall everything from Norton, Symantec, or McAfee, and just go to the MS Security Essentials website and use that instead. It's free, quiet, unobtrusive, and just works. That other stuff is just crap.

That's all I got. Hopefully this is helpful to some folks out there!

Sparky's back!

Maybe you've noticed that I'm smiling a lot more these days -- Could it be because Sparky's back?

I told him (and everyone) that things would have been a lot easier if we had met after about six months on my own. It turns out, I was wrong; I only needed three. Yay, Sparky!

We just got back from two weeks of fun, including family time in Denver, a visit from my sister and a side trip with her to Yosemite, and then almost a week in Provincetown and Boston. It was way more fun than any two people deserve to have!!

Now that I'm back I need to hunker down and get some work done, but I'll try and get some posts up about the trip real soon now.

The fat makes you look fat, Peg

Is it possible that my love handles look bigger now that I've lost ten pounds, because fat in easier-to-lose places has disappeared? Or are my love handles actually bigger now?

I hate you, mirror.

Still Getting Things Done

Wow, I finally caved in and dealt with the piles of paper accumulated on my desk at work today. I moved everything into one giant "inbox" which was about two feet tall, no exaggeration. And then went through EVERYTHING and tossed, categorized, or filed every. last. piece.

I now have a completely clean desk, for the first time in at least a year.

And worth a chuckle: While throwing out old to-do lists, I found one from when I moved into my new apartment: 1)Buy flatscreen TV; 2)Buy wireless router; 3)Setup computer; 4)Test poppers & porn

Manzanita Franciscana

This past winter, they found a one-of-a-kind plant in the Presidio that was thought to be extinct. Today, gossip columnists in SF Gate complained that my agency spent "taxpayer dollars" to move it to a safer location.

"Money may not grow on trees, but it sure cost taxpayers a bundle to move a bush that was found growing in the path of the Doyle Drive rebuild."

I cannot tell you how much this irks me. Check out the original article in the same paper, describing the botanist's joy of finding a plant thought utterly extinct for 60 years, just growing by the side of the Golden Gate Bridge. Funny how just a few months can change people.

I hope that little bush finds a new friend and makes lots of baby bushes for our roadside. San Francisco is a better place for it, and Matier and Ross can just suck it.

Car Shedding

Today at work, we were talking about strategies to encourage "Car Shedding" — getting households that currently own cars to sell them. In SF this usually means either a two-car household going down to one, or a one-car household going completely car-free.

It wasn't lost on me or my coworkers that I am a prime candidate for car shedding behavior.

• I own a car;
• It's old (I've had it for eleven years);
• I don't have a parking space at home
• I don't commute with it, since I usually bike to work and otherwise walk or take Muni;
• Both ZipCar and City Car Share have pods within just a few blocks of my house.

I constantly have to shuffle Kitty around to avoid street cleaning tickets. I even think that more than 50% of the times that I start her engine, it is to merely move her a few blocks to avoid a street cleaning ticket.

So, why do I still have my car? Yes, it was paid off years ago and I'm only paying insurance, gas, and maintenance. But seriously, on most days it's more hassle to worry about street cleaning when I get home at night than the benefit of being able to pick up my dry cleaning on Saturday without first reserving a ZipCar online (which takes about 30 seconds).

So, why do I still have my car? When I move later this year, I'm going to reconsider this.

Getting closer!

In my continuing saga to find a new place to hang my hat, I finally found a spot this past weekend that almost fit the bill.

Granted, I really wanted a view, a top-floor flat, and a deck, but apparently those things are way out of my price range. What I found instead was a flat in a new-ish building on the best street in the Mission (just down from LaLo, Mission Minis, and Boogaloos, for those of you who know 22nd Street), with a great kitchen and fabulous clearly-designed-by-a-gay-man bathrooms.

Looks like this one isn't going to work out, but after three months of househunting, to finally find something almost within my reach — with almost everything I wanted — is almost something to be happy about!

Pants pants pants

Today was the third day that I had to use the next hole in my belt; the first couple times I figured it was just a fluke. Now I think it's for real: I'm back to a 34" waist. I have no illusions that I'll ever hit 32" again, but this is a big improvement. Even my 36's were getting awfully tight this winter.

So, time for new pants! I've lost 10 pounds in six weeks, which I hope is slow and steady enough that my metabolism will keep the pounds off as long as I continue the exercise. I just wonder if I should keep these fat pants around in case I lapse. Crossing fingers.

Bay to Breakers 2010

So a funny thing happened Wednesday night. I flew back from the big conference in Arizona, only to realize that I had left my bike at Sparky's house the weekend before. I had to pick it up immediately, because a whole gaggle of coworkers had agreed to meet me Thursday morning for the annual BIKE TO WORK DAY. I convinced them all to go to my favorite diner "Boogaloos" at 8am for early breakfast and mimosas before we all biked into work together. (I know... mimosas before the office, but hey it's only once a year!)

The problem was, I had already told a friend I would meet him out that night to hear his DJ set straight from the Belgian clubs circa 1988-1990. Serious narrowcasting, yes, but it was his gig and I did actually enjoy that music the first time around, so I figured "why not?". Michalis (a friend from work) came with, and Sparky tagged along as well. We got to drinking... and then all of the sudden we had agreed to run this damn 12k race together that Sunday. As in, a 12k race in just four days with no training. Of course I had no intention of actually running, I just thought we were drunkplanning. As in, notplanning.

Bay to Breakers is a 7.5 mile (12k) race from the downtown SF bay waterfront all the way to the ocean breakers, hence the race name. This was the third time I've run the race; neither Sparky nor Michalis had ever run it. It's more of a street party than a race; it's more an excuse to drink early on a Sunday morning, in a tiger costume or perhaps completely naked, than a real run. It's a bit of a San Francisco tradition, too, this being the 99th year of the race!

Anyhooo, I've been running like a fiend at the gym since Sparky and I broke up at the beginning of April, so I didn't actually think this was going to be a tough race. And lo and behold, all three of us managed to run the entire race without stopping or walking, although we did lose track of Michalis at the beginning but found him again on one of the curves.

It was a great day. After the race we had burgers and beers in the Sunset, and other than Sparky losing his house keys somewhere along the way, I think we managed to pull off the whole spectacle without a hitch.

I can't wait for the 100th annual race, next May. Time to start planning my ouchfit...

Phoenix Liveblogging

Welcome to Arizona! I've heard Phoenix is like Los Angeles without Hollywood or the beach. So far... well, it's not quite THAT bad.

I arrived yesterday, here for four days to present some transportation research at the "Innovations in Travel Modeling" conference. My first experience was in the cab ride to the conference hotel, and the cabbie said he hates it here and is moving to Chicago next week. According to him, more than 40 conferences have cancelled due to the draconian anti-immigrant law that was passed last week, and he can't get any business. I was his third fare in 14 hours. He may have been exaggerating about the fare, but he was definitely leaving town.

Yesterday afternoon was the only free time I had here before the conference got underway, so I decided to sample the local flora and fauna by walking to the one gay bar close the hotel... well, "close" is relative because it was about a two-mile walk. But still, there are indeed sidewalks in Phoenix, a big step up from Houston. The place was nice enough; a bit like the Lone Star in SF with very friendly attitude-free guys. That's always a plus. In fact, I have to say I'm a bit surprised that the bar was actually fun.

Today's presentation on our bike route-choice model went fantastic, and we got lots of comments and positive reaction. I hope some other cities pick up on the research we did and take it further. Tomorrow is our downtown parking price research, and then Wednesday one last preso on advanced traffic analysis methods ("DTA") -- after which I can finally come home.

So far, so good! More to report soon.