When you do nothing, something always happens. Sometimes what happens is a negative thing. Just don't change the oil in your car or mow your lawn. At other times, the act of not doing something is beneficial.
I often see guys in the gym flitting from one exercise to another with hardly any rest between exercises or even sets of the same exercise. In effect, they've turned their workout into a cardio session. I used to do this myself. The bottom line is that when you're in the gym, your goal should be to build muscle (hypertrophy). The more weight you can lift with proper form, the more muscle growth you're going to ultimately stimulate. The problem comes when we jump into our next set before we've had the chance to fully regenerate our strength back to its full potential from the previous one. If the amount of rest between sets that you're currently doing means that you end up lilting less weight in the end or performing fewer exercises with the same weight, then you should be resting longer assuming that you want to squeeze out as much as possible during your gym sessions. For the average natural lifter like me, longer rest times allow you to move more weight across the workout as a whole.
Of course, there's no pre-set rest time for every individual lifter. I would just say that between sets, make sure you rest as long as you need to in order to feel fully physically and mentally recovered so that you can give maximum effort on your next set. This means waiting until your breathing and heart rate are down to normal. The after-effects from the previous exercise should have significantly subsided, and you feel able to lift again at your full potential. For many lifts that we perform, this could mean 3 minutes all the way up to 4 or 5 minutes or more. If you're moving through your workout at a faster pace, that's going to obviously increase the risk of injury.
This is probably not too big a deal for those smaller isolation exercises like dumbbell curls. But whenever you're moving more total weight, and if your goal is to train in close proximity to failure, then taking slightly longer rest times between sets improves your ability to achieve progressive overload in the big picture. Here I am today resting between sets of incline dumbbell chest presses. I'm resting for at least 5-6 minutes between sets to ensure that my technique remains as solid as possible.
Again, this is an instance where "when you do nothing, something always happens" is to be taken in a positive sense. There's a time to lift and a time to rest.