yeah, this is frustrating bug: if you log off when you're in a mine with your slaves working, they'll disappear; however, they will reappear as your followers once you go to the worldmap. If you enter the location to check the slaves, you'll see them. But then as soon as you leave the location, you'll "leave them back". If you "leave them back" on a worldmap, you'll never be able to get them back, but they still will be in your party.
To avoid this, one should speak with his slaves and order them to follow after he notice them following him on the worldmap.