I'd say your assumptions are correct. I'm betting your clutch isn't fully releasing which is putting some drag on your belt causing the hard shift.
Did you take the primary off? If so, you may need to check belt alignment as well.
Easiest way to tell is put your old weights and spring back in. if the problem goes away, then you know what caused it. Also may not hurt to check the condition of your clutch buttons/rollers. Blowing the dust out of the clutch can also help operation and longevity.
If you keep running it with the dragging belt, not only is it going to stay hard to shift, but the belt will eat away at the clutch face in time, and then you'll be buying a new clutch.