So, I have a big lot and I own properties adjoining on 3 sides, BUT there's one (crack) house that's adjoining on the back and they won't let me build my fence there. 

Last time I called the police and he stated that he would accept a survey. So, I got a full survey done and today the surveyor came back and set the pins and pink flags. I took a picture and later caught him kicking the flag down , he was again insisting that I would take part of their yard (actually, me fencing it off would put a cramp into their crack traffic, which cuts through another yard and then through mine to get to the back of theirs.). 

I had the police there again and they say that I'd have to take him to court, since this is a civil issue. But since I have all the proof that that is my property, wouldn't he have to be the one to take me to court, if he doesn't like the fence? He doesn't own my property. 

To make it more complicated, he doesn't own 'his' property either. It was his aunt's, who died in 2012. He is one of her many nephews and nieces, who took over the house and it's a hub for crack and prostitution. 

How would you deal with this? I have to build this fence to protect myself from them.

Build the fence if you are sure it is your property, if they don't like it they can sue and lose because it isn't on their land, if they tear it down or damage it you can sue in small claims court if you want but it may not be worth your time to do so.  I might also recommend a security camera to catch people trespassing and if/ when they tear down or damage the fence.

You will have a lot of trouble with them, find a good mason that works fast, I say mason because any wood, metal or plastic fence will be destroyed over night, build a block wall and fill it solid with concrete and rebar.

The problem is that we can't build the fence without them coming out. They don't go to work. There're always several people home 24/7. I don't know if I'll have to hire security guards to stand when we build the fence.

Originally posted by @Patrick Liska :

You will have a lot of trouble with them, find a good mason that works fast, I say mason because any wood, metal or plastic fence will be destroyed over night, build a block wall and fill it solid with concrete and rebar.

 Good idea, but we can't even be at the fence line without them challenging us.

A normal fence is usually required to back a few inches from the property line, if you had it surveyed and know exactly where that line is, then build it the required distance back, usually it's about 4", so give them the 4". let them challenge you, that's why i said a fast mason, have the wall built then fight them legally if they challenge, at that point it will be up. you have the proof of the property line, let them get there own survey to fight you.

Nobody is willing to do the work. There's no way to do it without them coming out at the time. This is a crack house and I'm sure there are a few guns in the house.

Could you hire an off duty police officer to be there while the contractor builds the fence?

Originally posted by @Bryan Bannach :

Could you hire an off duty police officer to be there while the contractor builds the fence?

 Brian, good idea. I was just thinking security guy, but an off-duty police officer would be much better. Would be money well spent. 

They will usually wear their badge, gun, etc. My company hires them to control traffic out of our complex.

I would completely agree about hiring the off duty officer.  Definitely money well spent.  Also, perhaps if the officer has a chance to see for himself the situation, he may take more of a vested interest in the problem, resulting in some special attention from the Street Crimes Unit (or whatever they call it in ATL).  

Regardless, even once you get the wall built, I wouldn't let up with complaints to the police until some results are achieved.  That saying about the squeaky wheel getting the grease is very true when it comes to problems like this.  If the police department gets enough complaints, especially if these complaints go high enough up the chain of command, you WILL see some action taken.  Having a crack house behind your property isn't exactly a value-add scenario...Good luck!

When I first started doing lending in Detroit.. this was an issue ... my client a big time flipper rehabber always had retired cops there with a clip board to check in the workers.. it was determined early on that much of the theft came from the exact guys that put the stuff in during the day..

they had the codes they knew what was there they would come in at night and steal it .

the day cop and clip board put that to bed.

So I agree with @Bryan Bannach   hire a retired policemen have him there during construction hours its the cost of owing property in the hood.. nothing unique about this.

or just sell the property and buy something else.. but I think your looking to assemble so I know you probably don't want to sell.

so just as stated hire security and I really like the idea of the @Patrick Liska   Cinder block fence I would for sure do that.. then put broken bottles in the concrete on top.. Just like you do in prisions that's what they know anyway.

There is no way I would waste my money or energy building a fence. No point in putting a band aid on a amputated limb. 

I would concentrate my efforts on driving out the crack house. It can be done, it is actually not that difficult, but there will be some risks and set backs along the way. If you do not get them out then don't waste your money on a fence...that will start a battle  you can not win.