The 2 bikes I had similar issues, it was a lean carb. While idling, can you quickly throttle up w/o hesitation?
If you put the choke on, does it still do it?
In both cases (XJ650), my idle mix was too lean and in one case, I needed to bump up the pilot jet. One of the bikes just took more time...