Contents
High availability Enterprise
Run two or three team servers in different zones. Each agent reports to all of them. When one server is down, the others still answer.
How it works
The servers do not talk to each other, and they need no shared database. Each agent sends its report to every server at the same time and combines the answers. The same warning from two servers is shown once.
How complete each server's view is
A server knows an agent only from that agent's last report to it. When every server is up, every server has every report. When a server misses reports, its view is incomplete until each agent's next step.
| Situation | Effect on warnings |
|---|---|
| A server was down or unreachable | It does not know the changes made meanwhile. Agents still get them from the other servers, because they combine all answers. |
| A server just restarted without its volume | It starts empty and fills up as agents take their next step, usually within a minute. |
| An agent's machine cannot reach some server | That server does not see this agent. Agents that reach only that server miss warnings about it until the link is back. |
| An agent stopped working | Its last report counts for 6 hours, then it is ignored. |
Warnings are therefore as fresh as the last step of each agent, and complete only as long as each machine reaches at least one server that the others also reach.
When a server does not answer within one second, the agent continues with the others. It then skips that server for one minute, so a server that is down costs the agent no more time. After a restart, a server catches up as each agent takes its next step.
Set it up
- Run two or three servers in different zones, with the same signing key and licence. The Helm chart does this by default. With
deploy.sh, useCOUNT=2. - Give each server its own address, for example
gitta-1.example.comandgitta-2.example.com. Do not put the servers behind one load-balanced address. - Connect repositories with all addresses.
$ gitta init --server https://gitta-1.example.com,https://gitta-2.example.com --token gt1_…
What to expect when a server is down
| Situation | Result |
|---|---|
| One of two servers is down | Agents get every warning from the other server. The first step on each machine waits up to one second. |
| All servers are down | Agents get warnings about their own machine only. Audit events wait on the machine and are sent when a server is back. |
| A server restarts | It rebuilds the team's state from the next report of each agent. With a volume, its team view and audit copies stay. |
Revoking tokens on several servers
A personal token works on every server that has the same signing key. Each server keeps its own list of revoked tokens.
gitta team token revoke tells every server and fails if it could not reach one of them. Until it reaches that server, the token still works there. Run the command again when the server is back, and check with gitta team token list. If a token leaked, also consider rotating the signing key, which invalidates every personal token at once.The Helm chart for Kubernetes runs two team servers by default. For how people and runners get tokens, see Tokens and sign-in.