To prevent that from happening you might want to slow down just a bit or move slightly backwards once the pole start moving in an unwanted direction. It is possible to get larger objects and buildings fall in even if your hole is not big enough: simply place yourself under one of the edges of the object for it to start leaning into the singularity and then move slowly towards its other end gradually consuming the rest.
