The biggest drawback is that you don't choose the guardian/conservator. Instead, the court chooses, and in most cases it is simply the first person who applies.
The process of appointing a guardian or conservator is quite costly in terms of attorneys' fees and administrative expenses, and the administration of a guardianship or conservatorship is complicated. Once the appointment process has ended, the guardian or conservator must file an inventory of assets, keep meticulous records, get permission before certain expenditures, and prepare yearly accountings for the court.
Finally, when the court appoints a guardian or conservator, the proceeding has the effect of determining that you are incapacitated. Unlike giving someone a power of attorney, this determination takes away your right to act on your behalf.
Your power of attorney can be either very broad or very limited. It can be drawn to take effect right away or take effect only if you become incapacitated. If the power of attorney is drawn to take effect only when you become incapacitated, you should deliver the power to your attorney in fact along with a letter explaining the circumstances under which you wish him or her to act on your behalf.