2201-2250 of 10000 results (37ms)
2021-10-05 §
11:28 <hnowlan@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase2023.codfw.wmnet: Switching over to using FQDN certificate - hnowlan@cumin1001 [production]
11:17 <hnowlan_> disabling puppet on cassandra nodes for rollout of 724061 - defaulting to cn=fqdn certificates [production]
11:15 <effie> upgrade scap to 4.0.2 - T291095 [production]
11:12 <urbanecm@deploy1002> Synchronized dblists/commonsuploads.dblist: 04524992865b0ae5750eb6fb0a374aa74a65b383: Enable local uploads for tcywiki (T166763) (duration: 00m 59s) [production]
10:11 <vgutierrez> update acme-chief to version 0.32 on acmechief hosts - T290249 [production]
10:09 <vgutierrez> update acme-chief to version 0.32 on acmechief-test hosts - T290249 [production]
10:06 <vgutierrez> upload acme-chief 0.32 to apt.wm.o (buster) - T290249 [production]
09:46 <hnowlan_> generated cassandra certificate using FQDN for restbase2023 [production]
09:09 <topranks> updating routinator on rpki2001 (T291543) [production]
08:59 <dcausse> depool and restart blazegraph on wdqs1007 [production]
08:51 <moritzm> installing openssl security updates for stretch (buster/bullseye already fixed) [production]
07:58 <moritzm> installing apache security updates [production]
07:57 <elukey> upgrade GPU drivers (AMD ROCm 4.3.1) on an-worker1[096-101] [production]
07:27 <ladsgroup@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-main' for release 'production' . [production]
07:26 <ladsgroup@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-main' for release 'production' . [production]
07:26 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=wdqs1004.wmnet [production]
06:38 <elukey> reboot an-worker1096 after installing new GPU drivers [production]
04:20 <eileen> civicrm revision changed from d74e9aa0a1 to 34d3c3aae8, config revision is cae09f7691 [production]
2021-10-04 §
23:30 <foks> resetting some emails used for abuse by a globally-banned user [production]
23:19 <jdrewniak@deploy1002> Synchronized portals: Wikimedia Portals Update: [[gerrit:726084| Bumping portals to master (T128546)]] (duration: 00m 59s) [production]
23:18 <jdrewniak@deploy1002> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:726084| Bumping portals to master (T128546)]] (duration: 00m 59s) [production]
23:13 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 75645c9cc59b37dbf59942eabbc014b7dc147626: Add explicit config for licensing/copyright message overrides (T284097) (duration: 00m 59s) [production]
23:05 <mutante> [deneb:~] $ sudo systemctl start docker-reporter-releng-images [production]
22:54 <mutante> puppetmaster2001 - rm /etc/logrotate.d/geoipupdate_ipinfo and geoipupdate_ipinfo ; running puppet, starting logrotate service [production]
18:13 <jiji@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
16:51 <bblack> rolling restart of haproxy for DoTLS on dns300[12],authdns1001,authdns2001 to recycle connections [production]
15:24 <vgutierrez> pool cp5006 [production]
15:17 <ladsgroup@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
15:16 <ladsgroup@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
14:50 <phuedx> phuedx@mwmaint1002:~$ mwscript extensions/SecurePoll/cli/purgeDecryptionKeys.php --wiki=votewiki --before="20210101000000" [production]
14:46 <ladsgroup@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
14:46 <effie> uploading scap 4.0.2 - T291095 [production]
14:45 <ladsgroup@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
14:39 <brennen> gitlab: upgrade to 14.3.2 (note there was an additional patch release on 2021-10-01) complete (T292256) [production]
14:25 <Amir1> cleaning up wb_changes_subscription rows from closed wikis (T292440) [production]
14:24 <brennen> gitlab: downtime for upgrade to 14.3.1 [production]
14:19 <elukey> import AMD ROCm 4.3.1 packages in buster-wikimedia's thirdparty/amd-rocm431 - T287267 [production]
14:19 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
14:17 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
14:13 <ladsgroup@deploy1002> Synchronized wmf-config/Wikibase.php: Config: [[gerrit:725905|Explicitly enable dispatching and pruning for wikidata (T48643)]] (duration: 00m 58s) [production]
14:08 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
14:06 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
14:03 <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gitlab2001.wikimedia.org with reason: upgrade gitlab2001 to new version https://phabricator.wikmiedia.org/T292256 [production]
14:03 <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on gitlab2001.wikimedia.org with reason: upgrade gitlab2001 to new version https://phabricator.wikmiedia.org/T292256 [production]
14:01 <ladsgroup@deploy1002> Synchronized wmf-config: Config: [[gerrit:725502|Enable dispatching via jobs everywhere (T48643)]] (duration: 01m 00s) [production]
12:57 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
12:56 <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:725785|Enable dispatching for wikidatawiki and commonswiki (T292088)]] (duration: 01m 00s) [production]
12:54 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
12:08 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
12:02 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti2026.codfw.wmnet with reason: Ganeti tests [production]