The consequences of such can be quite damaging. For example, a court can, inter alia, deduct the trustee's attorney fees from the beneficiary's portion of the trust. See Estate of Ivey (1994) 22 Cal.App.4th 873. Since lawyers are well-compensated this amount can easily reach the tens of thousands of dollars.
While this can be seen as a deterrent for a beneficiary interested in filing suit, it should not be a significant deterrent. A prudent trustee will hire an attorney to litigate the matter because they are ill-equipped to handle the rigors of litigation. While the requirements for filing suit are not incredibly intricate, some degree of clarity and precision is required. Unfortunately a petition filed by a non-attorney is likely to have errors in it, e.g. improper formatting, improper citations, etc. From personal experience, many probate petitions by non-attorneys I have read were lacking.
The California Professional Conduct rules, which apply to all California attorneys, require that the attorney not pursue a frivolous suit. See Cal Rules of Prof Cond 3-310. If a beneficiary then hires an attorney to litigate the matter, they should take solace in the fact that the attorney also faces consequences for filing a frivolous suit. Alternatively stated, the attorney too has skin in the game. Violation of an ethical rule can lead to professional discipline such as suspension or disbarment. Thus, the beneficiary should reasonably believe that the attorney will file suit only if the case has some merit to it. If the attorney deems the suit as being frivolous, that should give the beneficiary a potential early indication of how a judge might view the matter.
Still, eventual success or failure does not ultimately determine whether a suit is frivolous or not. The focal point is whether there was some shred of evidence for the suit. A suit involving a close-call is not frivolous. For example, a beneficiary sues a trustee for failing to sell oil company stock given the recent collapse in oil prices. The beneficiary could cite financial analysts who think crude oil has not bottomed out yet. In turn, the trustee could defend their decision and cite financial analysts who think oil prices will rebound next year. Regardless of the outcome, there are arguments for both sides. Hence, the notion of the suit being frivolous would not be present.
