The Abyss Stones are used to control their respective abysses. The stones were taken by the Third Overlord, when he went into the abysses to conquer them.
Every abyss has an abyss stone (except the Infernal Abyss). You will need all four of them to open the abyss gate to the Infernal Abyss in the Ruborian Desert. The abyss stones only appear in Overlord: Raising Hell.