We found something which could have been causing this. Next patch should not have this problem.
But let us know in case it still happens.

We are well disposed, this issue was introduced between two versions of the game where we had no proper autofix for it.

Can you simply put an earth wall at the tile marked red:

and then replace it with a claimed Empire tile again as it was before.
This should fix the entire problem.

The tile at Patrol point 4 is unwalkable for faction 3 and 4.
We still need to find out why this is. 

The only things that do change walkability for a certain faction are locked doors and toggled impasse flags.

Was maybe anything like this placed at this tile (Patrol point 4) before?

I spawned quite many beasts but the issue doesn't happen for me.
Maybe you have very low framwrate and they might do that because of too large time steps?

We did meanwhile increase the priority for collecting gold but also found a bug which made workers not collect gold, when there were more important tasks to do, but those had a too high threat so workers discarded them and did nothing.

This is all improved by now and so should be this issue.