Normally they would warn you that you are getting a Blu-Ray disk, the video rental store should have them clearly marked somehow. But cashiers might tend to forget since the disks will run on gaming consoles that many people already have.
They will exchange it for a DVD for you, and quite possibly refund any cost difference. In my local store it is $2.00 more to rent a Blu-Ray version.