The mistake was going back, some are small, hard to ID, but anything that even resembles a bunch of the same insects bumbling around, run fer the hills !!! LOL!
I have those in an unused chimney flue, no nest, concealed somewhere though. Act the same way when disturbed, when I saw the first few, got away from the edge and watched, they stay local mostly, go back, you will get nailed most likely.
Years ago, friend I worked for running a D3, had a huge lot job, just clear and grub, and I went at it, mostly done, hit a heavily populated one, lot of them all over, but they mistakenly went for the dozer and not me, I crept away, worked on the other side, when I came back through they had dissipated, open station, OROPS, that would have hurt, they tag you with pheromone, then swarm you.
The little ones, smaller nests and populations, seem to bumble more than find you, I stepped on a young or early nest while trimming, sting was not bad for what they can feel like, not that many around, and not aggressive, don't see those much any more, usually a hole in the grass or along a cut in the soil, I put a cover over the hole to see the traffic back up, they get agitated for a short while, then find a detour and act normal but the traffic does back up initially, funny to watch them deal with it, they always figure out a way to get in.
Worst thing I can think of is white faced hornets, I am very cautious when clearing, got face to face with one of the largest/mature nests I have ever seen, glad I left that bush alone, they nest in the strangest places, basketball size and then some, maybe 2, productive one for sure. Tractor and mower did not bother them, I mowed around the bush, but had I bumped any part of it, hard to even imagine how bad that would have ended, might get away, as if they don't target you, you have a window, linger, you're done for sure.