An internal gluster cluster (used by TC team) was put into a wildly broken state. The procedure to get it into the broken state is relatively unimportant, however, it exposed 3 particular problems. "gluster.peer.status" API is a little too restrictive, gluster has a VERY deceiving way of informing you the connectivity status of a peer, and, finally, the official 3rd party python library is omitting very important status information about the peers in the TSP. Looking at the below output should make it clear on what I mean by deceiving.

{
"uuid": "7bf52adc-9273-4468-a5c8-ce74391753db",
"hostname": "viper21.lab.ixsystems.com",
"connected": "Connected",
"state": "6",
"status": "Peer Rejected"
},
{
"uuid": "a2bdd4d5-05bb-4ab6-8bd5-bde857a2d167",
"hostname": "10.215.1.152",
"connected": "Connected",
"state": "3",
"status": "Peer in Cluster"
},
{
"uuid": "e65c9b35-6889-4d13-8a22-a1e0b4c663d7",
"hostname": "10.215.1.134",
"connected": "Disconnected",
"state": "3",
"status": "Peer in Cluster"
}

Furthermore, the official 3rd party gluster python library doesn't include the "state" or "status" keys. I've had to add those myself which is important since without them a peer can be "Connected" but "Peer Rejected" (see above)....which means it's not "healthy" in the TSP.