There's a lot of ugly hacks here, but this has been the state of prod
for months now, so we should reflect that.
Also, this bumps a bunch of workspace deps.
Change-Id: I744e0d3aff27036cfed73416cf442c7d62444a8b
Reviewed-on: https://gerrit.hackerspace.pl/c/hscloud/+/1473
Reviewed-by: q3k <q3k@hackerspace.pl>
Building jq portably is annoying, and the way we were doing it (which we
iirc stole from some google project?) sucked. Let's use a Go jq clone
instead.
This is an alternative for 1535. jq is currently used only in one
script, which could really be replaced by a Go program, but let's keep
it simple for now.
Change-Id: Ie25dffadd545df143490f510e9b75a74adf81492
Reviewed-on: https://gerrit.hackerspace.pl/c/hscloud/+/1540
Reviewed-by: palid <palid@hackerspace.pl>
Those are far from the latest versions still, but this change should be
non-breaking.
Change-Id: Ieeb9d6b301184f46677d821fe8276391346d6285
Reviewed-on: https://gerrit.hackerspace.pl/c/hscloud/+/1459
Reviewed-by: q3k <q3k@hackerspace.pl>
This commit aims to increase the speed of hscloud rebuild process
by optimizing the behavior of Go dependency fetching routines.
Gazelle v0.25.0 introduced a new dependency resolution mode
where it does not reach out for external dependencies; instead,
it operates solely on what is contained inside the workspace.
Because static dependency resolution mode is now the default
behavior in go_repository() rules, we are also updating the
contents of //third_party/go/repositories.bzl.
Also, I changed some of the bigger Go dependencies to be downloaded
by a tarball fetch in order to speed up the rebuild process.
Other changes:
* Bump nixpkgs to a fresh snapshot
* Upgrade to Bazel v5
Change-Id: Icfe752411b3128bcd5b25fa28bb76bec45ae2f71
Reviewed-on: https://gerrit.hackerspace.pl/c/hscloud/+/1441
Reviewed-by: q3k <q3k@hackerspace.pl>
You can test this using:
bazel run '@io_filippo_age//cmd/age'
The same target can now be used in data dependencies for secretstore
(you'll need to hardcode the runfile path, or use some
Bazel-runfile-resolving library for Python).
This required adding a few dependencies to
third_party/go/repositories.bzl, but also moving golang.org/x/crypto
from that file into WORKSPACE, before gazelle_deps gets loaded (as the
version requested by gazelle_deps is too old). We also moved shlex that
shouldn't have been in WORKSPACE into third_party/go/repositories.bzl.
Otherwise, this was just a few small deps - bumped golang.org/x/crypto,
new golang.org/x/term, new filippo.io/edwards25519. Hooray low
dependency code.
Change-Id: I0e684d88efffde13a3b4e253860aabcb35a3c94d
Reviewed-on: https://gerrit.hackerspace.pl/c/hscloud/+/1158
Reviewed-by: patryk <patryk@hackerspace.pl>
This switches over to rules_python's new pip_parse remote dependency
fetching, which significantly reduces Python hell in Bazel.
Now each Python dependency gets its own external repository, which means
we don't have to build psycopg on every hscloud checkout!
Change-Id: Icc3b39197fae1046648d9a483876f5de5bd415d0
Instead of manually packaging leaflet.js into the Git repository, this
uses an http_archive to download it on demand, and augments the static
serving code to accept different regexes as paths so that the
http_archive's contents can be served directly.
Change-Id: Icb8d624fea855fb748f107471133ac8adb5f2776
Gerrit 3.3.1 seems to have introduced a bug which makes the reviewers
column in the dashboard entry: https://bugs.chromium.org/p/gerrit/issues/detail?id=13899
This adds an override of gerrit.war to our Docker containers. The .war
is pulled over HTTP. It has been manually built by q3k from a source
checkout. The details on how this was done are in the WORKSPACE
http_file archive.
Once 3.3.3 lands we should get rid of it.
Change-Id: I8b64103cb87d8b185ff35165695a18cb19fea523
This is an attempt to see how well we do without rules_nixpkgs.
rules_nixpkgs has the following problems:
- complicates our build system significantly (generated external
repository indirection for picking local/nix python and go)
- creates builds that cannot run on production (as they are tainted by
/nix/store libraries)
- is not a full solution to the bazel hermeticity problem anyway, and
we'll have to tackle that some other way (eg. by introducing proper
C++ cross-compilation toolchains and building everything from C,
including Python and Go)
Instead of rules_nixpkgs, we ship a shell.nix file, so NixOS users can
just:
jane@hacker:~/hscloud $ nix-shell
hscloud-build-chrootenv:jane@hacker:~/hscloud$ prodaccess
This shell.nix is in a way nicer, as it immediately gives you all tools
needed to access production straight away.
Change-Id: Ieceb5ae0fb4d32e87301e5c99416379cedc900c5
per bazel error message:
DEBUG: Rule X indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = Y
Change-Id: I1c29609197d776536b7bc0336858047d7494d795
The new Go release fixes the following security issues:
CVE-2020-28362, CVE-2020-28367, CVE-2020-28366
See https://groups.google.com/g/golang-announce/c/NpBGTTmKzpM for more
detail.
The next step will be to re-build all our production Go binaries.
Change-Id: I97e4678a2e06a9559e66c0c0db008f0b6dfd81a2
This is an amalgamation of a handful of small changes to Go deps.
Notably:
- we remove our opencensus-proto fork, use upstream, use exclude=src to
fix the build
- unvendorify some deps
- bump io_rules_go to fix WKT resolution
Notably, we now do not have the 'protoc-gen-go' error when running
kubecfg/kubectl anymore.
Change-Id: I34fb9e78b2b12e4543142183d601d01987076f32
This introduces Nix, the package manager, and nixpkgs, the package
collection, into hscloud's bazel build machinery.
There are two reasons behind this:
- on NixOS, it's painful or at least very difficult to run hscloud out
of the box. Especially with rules_go, that download a blob from the
Internet to get a Go toolchain, it just fails outright. This solves
this and allows hscloud to be used on NixOS.
- on non-NixOS platforms that still might have access to Nix this
allows to somewhat hermeticize the build. Notably, Python now comes
from nixpkgs, and is fabricobbled in a way that makes pip3_import
use Nix system dependencies for ncurses and libpq.
This has been tested to run ci_presubmit on NixOS 20.09pre and Gentoo
~amd64.
Change-Id: Ic16e4827cb52a05aea0df0eed84d80c5e9ae0e07
instead of Python packages
As usual with Python sadness, the @pydeps wheels are built on the bazel
host, so stuffing them inside a container_image (or py_image) will cause
new and unexpected kinds of misery.
Change-Id: Id4e4d53741cf2da367f01aa15c21c133c5cf0dba
per bazel warning
DEBUG: Rule 'com_apt_itude_rules_pip' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1564255337 -0400"
Change-Id: I6564e8325aa31bbd156ffdf85854f3f5459bd4df
This removes some old skylib/zlib deps, moves all Go repositories to a
separate file, and in general does a small cleanup pass on external
repository magic.
Change-Id: Ic9700ee4c40cdb2e5a68e4fc7c6b3e386a109f8a
This turns the existing script into a proper sh_binary, and injects
dependencies (kubectl and jq) as deps into it.
This change also pulls in BUILDfiles for jq, and a dep (oniguruma) into
//third_party, and adds buildable external repositories for them.
The jq/oniguruma BUILDfiles are lifted from
https://github.com/attilaolah/bazel-tools/.
Change-Id: If2e548bd60a8fd34e4f3be767ae59c6b2f2286d9
We're not using them for anything. Initially they were going to be used
for nixops, but nixops is not very good, so let's just drop them.
We still have a Nix dependency for clustercfg.py when provisioning
nodes, but rules_nix/nixpkgs in WORKSPACE were unrelated to that.
Change-Id: I28c249507d1be9c5dbbd1ee764deccd9ab038549
Also drive-by modify WORKSPACE to add required deps.
Also drive-by update deps in WORKSPACE.
Also drive-by remove old stackb/proto library from WORKSPACE (only used
in cccampix, which is dead, and stackb/proto should be replaceable by
the main grpc lib by this point).
Change-Id: I7ac7fe2237e859dc1c45bf41a016174ed8e9ee71
We turn the existing experimental BUILD file into a nicely abstracted
starlark rule generator, for generating both external repositories and
container build rules from a single source of truth.
We also add 0.18.22 (which we already pushed via :push_latest).
Change-Id: I521d6e5cb9447eaf6f237671b7ef07d621cd9c77
This is a small service for accessing git repos read-only over gRPC.
It's going to be used to allow hackdoc to render arbitrary versions of
hscloud.
Change-Id: Ib3c5eb5a8bc679e8062142e6fa30505d9550e2fa
This is hackdoc, a documentation rendering tool for monorepos.
This is the first code iteration, that can only serve from a local git
checkout.
The code is incomplete, and is WIP.
Change-Id: I68ef7a991191c1bb1b0fdd2a8d8353aba642e28f
