You have the idea right.
If your bike makes the noise primarily at idle until it warms up,,, it may help.
How many miles on the bike?
Carefully remove the top screw.
(Be careful when you do as a spring and small pin will jump out).
Put a flat ended punch or screwdriver in the hole.
It will rest against the adjuster when inserted.
Give the punch a sharp tap. (Don't bash the hell out of it).
If there is sufficient chain slack the adjuster will move to the next step and quieten the motor.
Re-install pin/spring top bolt, and see if it helped.
NOTE: All Connies have a certain amount of chain noise, I suggest you compare with others.
I suggest; join the club and come ride with us.
Then you'll have lots of Connies to compare with and I guarantee you'll like the group!
Ride safe, Ted