4 - 6k cals is a big range I reckon and without tracking it's hard to know if you're actually in a small surplus or going too much. Tracking even for a week or two could help. The thing here is if you’re clean with the food and monitor weight gain week to week you can definitely push cals a bit more without getting soft. You just need some structure