Identity server really REALLY REALLY wants you to do administration via HTTPS. Just realize, that if you try to circumvent this (for your local development) then the Identity Server will try very hard to convince you that it cannot be out-smarted. Be prepared to get pretty mad and frustrated as you try to out-think it.
So yeah, I changed the settings, so I didn’t have to use https. Login code seemed to work great. All of the settings (in code) said that I was logged in, but step-through debugging just WOULD NOT show the darn admin menu. I tried a lot of stuff and I did not find a way to trick this into working. Luckily, this is written in MVC. So it was much easier to simply test that the code was working, and then just leave it at that.
Solution: Don’t fight it. Just use HTTPS.