Please post you prolactin test results. If they are actually very high, then it would explain your docs concern and you may really need cabergoline. Not everything shows well on an MRI if it is tiny.
Note in the study that micro prolactinoma were producing ~182 iu/l prolactin and macro prolactinoma were ~2019. These numbers are in contrast to people on the forum posting "high prolactin" with values such as 20 - often rattling with fear that their nipples will leak. There is a lot of myth in bro science regarding using cabergoline (although it is possible some receive some symptom benefits to altering dopamine).
The rule on meds is this: There is no such thing as a safe med, but meds can be used safely.
Since you have been using androgens, that may have given you higher than normal readings. If your readings are above 100, then probably your doc is right. (Does he know of your androgen use?). If you truly need cabergoline, then it can be used safely. Most prolactinoma can be sucessfully treated with 3mg/wk for 2 years. Only a few need it for lifetime and at that dose, even lifetime use would make heart valve fibrosis unlikely.
Another rule with cabergoline (or any ergot): Never under any circumstances use an SSRI, SNRI or serotonin increasing medication.