Um, if Big Brother knows I'm drunk and being driven home safely, I'm okay with that. If I need to get to the hospital fast, I want emergency services to know. If a terrorist is tracked, so be it. If I plan a trip from SF to Grand Canyon, and I'm tracked as I cross state lines, who cares? I'm not important enough for the government to pay attention to. If they track me, they'll die of boredom.
If you are worried, don't use a driverless vehicle. Oh wait, you'll have to limit yourself to 1966 VW Bugs, because any new vehicles will track your movements whether the car pilots itself or you do.
There are advantages and disadvantages, it's true. My personal feeling is that the advantages far outweigh the disadvantages.