It sounds like plum sawfly larvae. Plums, like all fruit trees, are prone to a variety of insect and disease pests. When growing fruit trees, you need to commit to a regular spraying schedule such as that described on the label of Bonide Fruit Tree Spray. Here’s a link to more on the plum sawfly.