These sorts of things tend to be governed by state law rather than federal, so it's hard to give a definitive answer as to what your rights are in this instance. Chances are, he cannot place unusual restrictions on when you can pick up the check.
If you are expecting a reference of any sort, it doesn't pay to rock the boat. If you want to show up during business hours to make sure everyone knew how you were slighted, forget about it, it's not worth it.
If it's inconvenient for you to pick it up then, let him know, and maybe you can swing by at a designated time and he can meet you at the door or something. If you are afraid of showing up alone after hours, you can either bring a friend or two, or alternatively simply tell your former employer you are not comfortable with that arrangement (whether you want to admit your fear outright is up to you).