My previous boss requested me to pick up my last pay check (I quitted the job) when the company was closed just because he does not want other people know what he did to me.
I would not go after hours. The employer is supposed to make pay checks available during normal business hours. let him know you will be there during that time, if they refue to give you your check file a complaint with the labor board. Holding a paycheck unlawfully if against the law and employer will have to pay you interest on top of that.
These sorts of things tend to be governed by state law rather than federal, so it's hard to give a definitive answer as to what your rights are in this instance. Chances are, he cannot place unusual restrictions on when you can pick up the check.
If you are expecting a reference of any sort, it doesn't pay to rock the boat. If you want to show up during business hours to make sure everyone knew how you were slighted, forget about it, it's not worth it.
If it's inconvenient for you to pick it up then, let him know, and maybe you can swing by at a designated time and he can meet you at the door or something. If you are afraid of showing up alone after hours, you can either bring a friend or two, or alternatively simply tell your former employer you are not comfortable with that arrangement (whether you want to admit your fear outright is up to you).