If you ask me...
...the guy is full of it...you're probably disturbing his nappy-time or something...How long has he been around? Has this been an ongoing thing? If not, what changed...why now?
IMO all you need to do (really more for your protection) is show some courtesy and common sense...If people are passing by while you work, just stop 'til they pass...either that or get some orange cones and safety tape and establish a temporary work zone around your perimeter...or tell him there are umpteen things you would like not to happen for any number of reasons, but that's life and we all know why space is a vacuum...
jimHJJ(...because the world sux...)