Plants are very sophisticated when it comes to experiencing the world and communicating with each other and other creatures. From what little I have learned it appears they experience/communicate almost entirely through chemical signals (finding food, recognizing close genetic relatives, predators, friendly insects etc...).
If we actually touch the leaves it likely does have a strong effect, they probably recognize our chemical signature and decide whether we are friend or foe out to eat/harm them or beneficial.
I had to thin some seedlings in trays the other day and felt guilty about it, I also wondered if the plants saw me as an evil threat afterwards if they perceived me killing all of the other seedlings in the tray with scissors. Next time will scatter fewer seeds as it made me feel bad to cull the extras.
Here is a documentary on their experience and communication. At the
16:00 min mark it talks about wild tobacco and how sophisticated it's system of perceiving threats is.