How should one go about tuning hyper parameters?

I am building a model and I know that there are several hyper parameters that can be tuned. If I spend several days I.e tuning the learning rate and then find the optimal one, do I then use that learning rate set in stone and then explore tuning another hyper parameter?

Or is there a way to perform a huge search on 5-10 different parameters at the same time?

datascience go