I had the exact same issue as you. "within range" but still very low. I was 380 I think. Have you ever had trauma to your testes? Have you been on anti depressant medication for a long time? Have you had a bad diet all your life? Whats your sleep like? Have you had symptoms of erectile disfunction, small amount of semen when you ejaculate, difficulty building muscle? I would assume yes to that last question. Your endo is an idiot for saying it was nothing to worry about.
You can do a PCT and see if that brings you up in the long term. Going on TRT isn't a big deal like it used to be but you should certainly avoid it if possible.
I would do pharma clomid and nolva for 2 months, wait another 2 months and go get tested again and see if theres any improvement.
I went from 380 to 680 just doing a PCT. I still decided to cycle, but I probably would have been able to build a decent amount of muscle at 680. PM me or post in here if you have any more questions bro. I'm here to offer my experience if its at all helpful.