Island of the Fishmen (1981)

After their prison ship sinks in the Caribbean, a group of prisoners and a doctor wash ashore on a seemingly deserted island. They soon discover a strange couple, who invite them to stay at their house. While the prisoners plan an escape, the doctor does some investigating, and soon finds out just what the pair are really doing, and why the prisoners keep disappearing mysteriously.