Pokeweed is both edible and poisonous depending the part of the plant. It is not surprising that the pollen might be irritating but it isn’t likely to poison you from the smell. To be safe, avoid all contact with the plants and stay away from it while it is in flower since you are having a reaction to the smell. Here are a couple of links you might find helpful:
http://www.mountsinai.org/health-library/poison/pokeweed-poisoning
http://www.illinoiswildflowers.info/weeds/plants/pokeweed.htm