We ran out of disk space on the old PVC. Made a new one, copied data
over, and this change points the postgres data mount to that new PVC.
Change-Id: Iea4e140680066a3335cc69caf9293093f90bb568
Previously: 856b216459 switched to using a
Secret instead of a ConfigMap for appservice-irc. That however didn't
update the bootstrap job which still used the ConfigMap. This fixes
that.
Change-Id: I50f33935691678ce24ecf4e04d7ce1b13c184929
Apart from this, we also had to manually edit the registration yaml to
add @libera_ and #libera_ prefixes to the allowlists.
Change-Id: If85f58cf3d1291e0bf9099ef13d9397040a47782
This doesn't have to be publicly reachable, as the future
//cluster/identd will dial into the pod directly to access the
appservice's identd.
Change-Id: I139341ead76309a6640eeb9a278462565290dd34
These contain a channel key for a secret channel.
We also had to migrate the appservice-irc config to a secret.
Change-Id: I92c7cdf9679f65d9e655e22d690cef2e83180135
This is the case for any IRC server that has ignoreIdleUsersOnStartup
set, because of what seems like an appservice-irc bug.
Change-Id: If5063a3bc2d79c7f2fc79ec7560bf9bfe2b25aba
This allows us to bypass the issue where Kubernetes jobs cannot be
updated once completed, so bumping appservice image versions was
painful.
But really, though, this is probably someting that kubecfg/kartongips
should handle.
Change-Id: I2778c5433f699db89120a3c44e55d2fbe2a10015
This allows people to save their NickServ passwords into bridge's
storage. Obviously nobody should trust us tho.
Change-Id: I2afe9e5215cd8f7419e9eab8183789df13e21aac
This should alleviate an issue of people getting joining and immediately
getting dropped off due to client limit on bridge restarts.
Change-Id: Ideb13ba9930d565ede728d2750d0c7af04746cf1
Newer versions of alpine edge repos have a `yq` that behaves oddly:
$ kubectl -n matrix-0x3c logs -f appservice-telegram-prod-85d66696c6-9drnl -c generate-config
+ apk add --no-cache yq
fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
(1/1) Installing yq (4.4.1-r0)
Executing busybox-1.31.1-r21.trigger
ERROR: busybox-1.31.1-r21.trigger: script exited with error 127
OK: 11 MiB in 15 packages
+ cp /config/config.yaml /data/config.yaml
+ yq r /registration/registration.yaml as_token
Error: unknown command "r" for "yq"
Run 'yq --help' for usage.
+ yq w -i /data/config.yaml appservice.as_token
Error: unknown command "w" for "yq"
Run 'yq --help' for usage.
This downgrades back to a working yq.
Change-Id: Ifc77bcc88156b02f3ec17e6f84c5615149108777
This is used by some external modules (appservices/instance
definitions). In order to reduce scope of (untested) changes in this
rollout, let's temporarily backport that function into matrix-ng.
Change-Id: Ib1054844391497ef1455b25c7f939c68c628ff09
matrix-ng split into multiple submodules causes some changes in keys
that might've been used for homeserver/riot configuration customization.
Migration to kube.Namespace.Contain has also caused change in Deployment
selectors (immutable fields), thus needing manual removal of these
first.
This is, as always, documented in lib/matrix-ng.libsonnet header.
Change-Id: I39a745ee27e3c55ec748818b9cf9b4e8ba1d2df5
This is a major revamp of our matrix/synapse deployment as a separate
.libsonnet module.
* synapse version bump to 1.25.0
* riot-web version bump to 1.7.18
* Replaced synapse migration hack we used to template configuration with
environment variable replacement done by Kubernetes itself
* Implemented support for OpenID Connect, migration from CAS has been
verified to be working with some additional configuration options
* Moved homeserver signing key into k8s secret, thus making it possible
to run synapse processes without a single data volume
* Split synapse into main process, generic worker and media repository
worker. (latter is the only container using data volume) Both generic
worker and media repository worker is running on a single replica, until
we get proper HTTP routing/loadbalancing
* Riot nginx.conf has been extracted into an external file loaded using
importstr.
Change-Id: I6c4d34bf41e148a302d1cbe725608a5aeb7b87ba
Exposes /.well-known/matrix/ metadata endpoints on cfg.webDomain that
are required for federation to work properly. This can be enabled using
cfg.wellKnown flag set to true.
Change-Id: I097b58efc7442b904a135d4519999e36d155c197
It reached the stage of being crapped out so much that the OSDs spurious
IOPS killed the performance of disks colocated on the same M610 RAID
controllers. This made etcd _very_ slow, to the point of churning
through re-elections due to timeouts.
etcd/apiserver latencies, observe the difference at ~15:38:
https://object.ceph-waw3.hswaw.net/q3k-personal/4fbe8d4cfc8193cad307d487371b4e44358b931a7494aa88aff50b13fae9983c.png
I moved gerrit/* and matrix/appservice-irc-freenode PVCs to ceph-waw3 by
hand. The rest were non-critical so I removed them, they can be
recovered from benji backups if needed.
Change-Id: Iffbe87aefc06d8324a82b958a579143b7dd9914c
This is in preparation for spinning up a staging/QA matrix instance,
where the MXID domain is under control by hscloud machinery (and not a
top-level organizational domain).
Change-Id: I10505615ebb407b3b2eac0c1b87ad5625e2009c0
This is in prepration for bringing up a Matrix server for hsp.sh.
Verified to cause no diff on prod.
Change-Id: Ied2de210692e3ddfdb1d3f37b12893b214c34b0b
This deploys office.hackerspace.pl. It's a collaborative document
editing server that works with Nextcloud.
This is already live, and can be tested with owncloud.hackerspace.pl
(new -> document).
Change-Id: Ic8055a8a6679e7a0695ebb9e41108074d8f789af
WHITE
WHALE
HOLY
GRAIL
Complex systems are complex. Let me tell you a story about that.
Matrix clients perform their last stage of login by performing a POST to
/_matrix/client/r0/login on the Matrix homeserver they log in to. How
they reach the Homeserver is specified earlier - either by using
discovery via SRV or .well-known, or by the client manually specifying
the Matrix homeserver URL.
Regardless of how they reach this endpoint in the first place, this POST
endpoint, as per the Matrix Client-Server API Specification (r0.6.1),
MAY return a `well_known` key, which MUST contain a `homeserver`
address, pointing to the address of the homeserver which the client
should talk to. If present, the client SHOULD use that instead of
whatever it connected to so far.
Issue the first: the iOS client requires `well_known` in that response,
and doesn't work otherwise. https://github.com/vector-im/element-ios/issues/3448
Issue the second: Synapse will return `well_known` accordingly, but only
if `public_baseurl` is set in its configuration. It is not required to
be set. If not set, it will simply not return this key.
Shrek the third: we never set `public_baseurl` in Synapse, and the first
issue (iOS needing `well_known`) only became a regression in
https://github.com/vector-im/element-ios/issues/2715 . As such, it was
difficult to troubleshoot this issue, and we kept getting on some red
herrings: is it the SSO? Is our server broken? Is the iOS implementation
broken?
But now we know - https://github.com/vector-im/element-ios/issues/2715
seems to be the true culprit.
Change-Id: I913792e31e3c6813d4e51d4befdba720cad3f532
Configuring this one is a bit different from appservice-irc. Notably,
there's no way to give it a registration.yaml to overlay on top of a
config, se we end up using an init container with yq to do that for us.
Also, I had to manually copy the regsitration.yaml in synapse, from
/appservices/telegram-prod/registration.yaml to
/data/appservices/telegram-prod.jsonnet, in order to make it work with
the synapse docker start magic. :/
Otherwise, this is deployed and seems to be working.
Change-Id: Id747a0e310221855556c1d280439376f0c4e5ed6
This is in preparation for adding a Telegram bridge appservice. The main
jsonnet file was getting quite chonky.
This does not affect production, and is just a refactor.
Change-Id: I7cdee2bd71aedb40a9f6c3e5148f829023171dcb
The way this was migrated is not to be spoken of.
(hint: it involved downtime, and mounting two volumes at once)
appservice-irc has some storage, we should migrate that to waw3, too. But
it's not as critical.
The new storage (waw3) is _much_ faster.
Change-Id: I4b4bd32e4fedc514753d25bac35d001e8a9c5f00
When deploying https://gerrit.hackerspace.pl/c/hscloud/+/401 we manually
re-pinned appservice-irc to run on bc01n03 (to prevent reschedule as
bc01n02 was updated while bc01n03 was already done). This change makes
git reflect production.
Change-Id: I2518a8a227bfacefd9f1905ded5a1d65e379845f
This has already been bumped in production, and this change makes it
reflect that.
This was supposed to fix iOS sign-in, but that didn't seem to have
worked.
Change-Id: I9278490e40b332a8439fdf1361f27df770b8cd9e
At some point someone bumped appservice-irc to 0.17.1 without commiting
this to git. This fixes that, and also drive-by refactors the
appservice-irc image version to live next to all the other version
strings.
`kubecfg diff --diff-strategy=subset prod.jsonnet` now shows no diff.
Change-Id: I90a64d05cc72669de41fa68195672adca2eb37e8
Only these nodes (and bc01n03( are #blesed by freenode.
In the future we should fix this by having custom node labels for
blessed nodes. But this will do for now.
Change-Id: Ia5d7cfcb9329da0de8d596ed40b20b0e0f286f43
Some pods got evicted. Some of them broke.
- postgres in matrix and nginx in internet because of the new policies
(chown issues)
- cas proxy in matrix because apparently the image was not reuploaded
to the regsitry after ceph-waw1 died, and another node didn't have it
- registry because it had a weak image pin an downgraded to some
broken version on another node
Change-Id: I836036872629843c8ede1b7f67982112c90d71f0
Dynamic config generation based on environment variables in Synapse is
no longer supported. To pass secrets to container we use a patch that
implements configuration overrides via environment variables directly.
(to be upstreamed...)
Due to Synapse update, appservice configuration ConfigMaps don't need to
be copied into Synapse /data volume anymore.
Change-Id: I70e6480983bfb997362739c6ce0ec3c313320836
This makes a registry be automatically part of the cluster
infrastructure.
Tested by running kubecfg diff, no diffs (apart from out-of-date ACLs)
found.
Change-Id: Ic0635e789cf3fb851f410bcf2865326f1fa87545
We have quite a bit of them at this point, and we're likely going to use
app/* and go/svc/* for 'core' services only anyway.
Change-Id: Ic315fbd2d672e525439992bfcd9ead730d1a1b71
Another change I lost somewhere in the process of remembering how to
gerrit.
I rewrote it (lost the original commit), and also added the (upcoming)
egressifier service.
Change-Id: I1647bc3b1e504a192150ab76f4c6d1709e608f0a
This change impelements the k8s machinery for Gerrit.
This might look somewhat complex at first, but the gist of it is:
- k8s mounts etc, git, cache, db, index as RW PVs
- k8s mounts a configmap containing gerrit.conf into an external
directory
- k8s mounts a secret containing secure.conf into an external directory
- on startup, gerrit's entrypoint will copy over {gerrit,secure}.conf
and start a small updater script that copies over gerrit.conf if
there's any change. This should, in theory, make gerrit reload its
config.
This is already running on production. You're probably looking at this
change through the instance deployed by itself :)
Change-Id: Ida9dff721c17cf4da7fb6ccbb54d2c4024672572
q3k
viq
informatic
Norbert Szulc
