Below the water pump pulley there is a "double set" of drive pulleys (actually one unit that has two different diameter drives). This is bolted to the face of the balancer. In the center of the pulleys there is a bolt and washer that secures the balancer to the crankshaft. That is the bolt that you need to check. If the balancer starts to "walk" off the crankshaft, the bolt will either be broken, loose, or missing. Another way to tell if the balancer is beginning to walk on the shaft is by looking to see if the main drive belt (the one that spins the P/S pump and alternator) is no longer riding in the center of the top belt tensioner. As the balancer slips toward the end of the crankshaft, this belt will tend to ride toward the front edge of the tensioner.
Unfortunately, if the balancer is slipping, you are better off replacing it now before it breaks. It is much easier to remove as a whole unit than to try to remove the sleeve that remains on the crankshaft after the balancer shears off.
But, like I said, this may have nothing to do with the symptoms you are noticing.