Tillräckligt nog har ett svar på detta nu tillhandahållits i filmen Rogue One.
Rogue One (släppt dec 2016) följer spoilers. De är MAJOR spoilers. Jag skojar inte.
The callsign "Rogue One" is improvised on the spot by Imperial defector Bodhi Rook as he makes an unauthorised departure from the Rebel base in a (previously stolen) Imperial Shuttle. He is with Jyn Erso and her strike team as they leave on a mission to steal the Death Star plans.
Although the Rebel launch controller complains at the time that the callsign doesn't exist, it seems evident that the Rebel leaders later adopted "Rogue" as an official designation in honour of Bodhi, Jyn, and the rest of the strike team who gave up their lives to retrieve the plans and give the Rebel Alliance a new hope of survival and eventual victory.
In the scene itself it is somewhat ambiguous where Bodhi takes his inspiration for the name from. He's looking at Jyn for guidance, and it's possible it's a direct reference to her. It's also possible he's referring more generally to the whole team going rogue on a mission/plan the Rebel leadership has just rejected. A third possibility is that he's referring to himself, having defected from being an Imperial cargo pilot and soon afterwards disobeying orders from the Rebel Alliance as well. It may be a mixture of these factors.
The timing of "Rogue" coming into use as an official callsign by the time of The Empire Strikes Back also fits in nicely. The events of A New Hope follow on so closely from Rogue One that the Rebels wouldn't have had time for much in the way of administrative changes — and the successful destruction of the Death Star is what really solidifies the crucial historical importance of the sacrifice made by Jyn, Bodhi, and the Rogue One strike team.