I agree, but it sounds like there is a problem with unauthorised activation on certain handsets. If this is the case, then Google need to get it sorted ASAP.
From Google's FAQ's:
"After Google Maps for mobile with Latitude is installed and running on some mobile devices, you may receive prompts on your device reminding you that you have enabled Latitude to share your location with selected friends. These reminders allow you to continue or stop sharing your location with Latitude and will appear a limited number of times if you have enabled Latitude but have not used it recently"
This is what Privacy International have picked up on, and all the fuss is over. It's basically saying that once the servce is activated, it can continue to run without the owner knowing. Where someone else has activated it, for example a nosy boss, then the owner is being spied on.