Legal Question in Family Law in California

That's not a ground for annulment.

Mr. Roach is correct, you have not mentioned anything that would be grounds for annulment of your marriage. Unlike divorce and legal separation, which can be had for essentially no reason other than you want one, there are strictly limited grounds on which an annulment can be had. If you do have grounds for annulment, however, and you have some reason to go that route rather than divorce, you most likely can avoid any issues such as you mention by declaring "putative spouse" status. Of course once you do that, then the law treats the annulment like a divorce in most respects, so it is somewhat circular.