Also, I noticed YouTube videos of elevator photographers getting busted by security after riding/filming elevators, even passenger ones. Of course the property owner always has the right to remove someone if they believe they are trespassing. Some people complain that photography is prohibited, I won’t get into detail here as of course there are places you should NEVER take pictures/videos in, but at the same time claim that photography is not a crime. Do you know if you had to deal with an incident like this or know someone that has?
The thing that people tend to forget is that security isn't generally making the decisions or policies by themselves. Every property has a universally-recognized industry paper called "Post Orders" that define the policies and procedures that security are to enforce. This defines what people are allowed and not allowed to do, how security is to respond to certain scenarios, a use of force policy (for public safety departments), building access policies, etc.
In my eyes, if I saw someone who I did not recognize to belong in the building (in the case of an office), pull out a camera and start filming the elevator, I would likely respond by monitoring their movements using CCTV. As long as they are not attempting to circumvent building security systems, and they aren't hopping off random floors and walking around near or in tenant spaces, I don't really have a reason to bother them. Now if they pull out an independent service key and try to bypass the card reader, then I have a problem that I have to respond to. The filmer might mean no harm, but they've just breached security. I have to respond to that, regardless of intent. At the very least, I am going to find the person and ask what they are doing and why they are using elevator keys. If they give me a reasonable explanation that they're a hobbyist trying to film an elevator, I might give them a warning to not do that again and let them continue. Barring that, they're getting ejected. It's nothing personal, but just a part of the job I was given to do.
The industry, like any other, has its undertrained personnel and "bad apples" that take it too seriously. It is correct that photography is not a crime. The common phrase "public property" is thrown around a lot in this community, and there's a misconception as to what it really means. Unless you're in a government building, walking up a city street, or at some other government funded facility like a park, chances are you're on private property. The term we're trained to use for buildings like shopping malls, hotels, etc, is "private property in which the public is invited." This means that by default, anyone can come; but at the discretion of the building owner, or their designated representative ("Security"), that welcoming can be revoked. Filming isn't a crime and nobody can tell you, legally, to delete what you've captured. However, if you refuse to leave when asked, you're now trespassing.
In general, if there are things that a building owner doesn't want someone to see, then they need to implement measures to ensure that only authorized personnel are allowed to see it. Everyone has cameras nowadays. If it's that important, they need to lock it behind a door and key. But if your building has unlocked doors and not even the simplest access control measures in place, then they shouldn't be surprised when someone comes exploring. It's human nature, after all.