Legal Question in Family Law in Georgia

I gather by the fact that you posted here instead of asking your lawyer that you made the one catastrophic mistake guaranteed to make a custody battle go badly - do it without a lawyer. Big mistake (and that is likely what went wrong).

As a general rule, representing an ex-brother-in-law is far too remote a connection to support recusal, but it's fact dependent, and your lawyer would have raised it at the time if it mattered (you can't wait a year and raise it).