Legal Question in Real Estate Law in California

It would depend on details of what was done on and with the portion of your property on the neighbor's side of the fence, and how the property was taxed.

You do not have a right to move the fence without the neighbor's consent or a court order.

In general, the law disapproves of "self help" to end trespasses or remove encroachments. One notable exception is the right to trim overhanging tree limbs or invading roots, but there is no similar exception for mis-located fences.

You will have to negotiate with the neighbor or take him to court, and as Mr. McCormick suggests, the neighbor may have defenses to being "de-fenced."