2151-2200 of 10000 results (42ms)
2021-06-05 §
12:11 <wm-bot> <lucaswerkmeister> ran extensions/WikiLambda/maintenance/reloadBuiltinData.php, most objects seemed to work, only a few errors (T284368) [tools.notwikilambda]
11:53 <wm-bot> <lucaswerkmeister> tried to run extensions/WikiLambda/development/updateLanguageCodes.php, most objects failed with the same “value en for Z11K1 … invalid” error as quoted in T284368 [tools.notwikilambda]
11:36 <wm-bot> <lucaswerkmeister> made wiki read-only for maintenance (T284368) [tools.notwikilambda]
00:21 <mutante> backup1001 - systemctl baclua-dir works again (restoring backup for non-existing host) [production]
00:18 <mutante> backup1001 systemctl reload bacula-dir fails [production]
2021-06-04 §
23:57 <Krinkle> integration-agent-qemu-1001 back up, Thanks andrewbogott [releng]
23:47 <Krinkle> Qemu jobs are stuck. Jenkins is unable to connect to integration-agent-qemu-1001 [releng]
23:25 <wm-bot> <lucaswerkmeister> deployed 16c0cd2606 (improve batch mode results page) [tools.lexeme-forms]
22:08 <cwhite@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host doh4001.wikimedia.org [production]
21:51 <cwhite@cumin1001> START - Cookbook sre.ganeti.makevm for new host doh4001.wikimedia.org [production]
21:30 <bstorm> deleting "tools-k8s-ingress-3", "tools-k8s-ingress-2", "tools-k8s-ingress-1" T264221 [tools]
21:21 <bstorm> cleared error state from 4 grid queues [tools]
20:59 <bblack> repool cp1087 - T278729 [production]
20:13 <James_F> Zuul: Switch almost all node10 jobs to node12 T284345 [releng]
20:11 <James_F> Zuul: [VisualEditor/VisualEditor] Switch node10 jobs to node12 T284345 [releng]
20:11 <bblack@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1087.eqiad.wmnet with reason: REIMAGE [production]
20:09 <bblack@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp1087.eqiad.wmnet with reason: REIMAGE [production]
19:19 <James_F> Docker: Publishing node12 CI images T284343 [releng]
19:06 <bblack> depool cp1087 - T278729 [production]
18:21 <ayounsi@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
17:42 <razzi> sudo cookbook sre.aqs.roll-restart aqs to deploy new mediawiki history snapshot [analytics]
17:36 <razzi@cumin1001> END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) [production]
17:33 <razzi@cumin1001> START - Cookbook sre.aqs.roll-restart [production]
17:33 <razzi@cumin1001> END (FAIL) - Cookbook sre.aqs.roll-restart (exit_code=99) [production]
17:33 <razzi@cumin1001> START - Cookbook sre.aqs.roll-restart [production]
17:28 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: REIMAGE [production]
17:25 <andrew@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: REIMAGE [production]
15:25 <topranks> Adding 1:1 NAT configuration for fran2001 / analytics.codfw.wikimedia.org to pfw3-codfw (backup site) [production]
15:00 <Lucas_WMDE> wb-reconcile: ran `git gc --prune=now` in mw and wb and switched WikibaseReconcileEdit back to main [wikidata-dev]
14:47 <krinkle@deploy1002> Synchronized wmf-config/CommonSettings.php: I434d9cfa29d84f (duration: 00m 56s) [production]
14:46 <krinkle@deploy1002> Synchronized php-1.37.0-wmf.7/extensions/DiscussionTools/extension.json: Iea41ab8599ffae (duration: 00m 56s) [production]
14:44 <krinkle@deploy1002> Synchronized php-1.37.0-wmf.7/extensions/DiscussionTools/includes/: Iea41ab8599ffae (duration: 00m 59s) [production]
14:41 <krinkle@deploy1002> Scap failed!: 9/9 canaries failed their endpoint checks(https://en.wikipedia.org) [production]
13:39 <Krinkle> mwmaint1002: Running purge_parsercache_now.php on pc1008, server 3/4, ref T282761 [production]
13:33 <oblivian@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
12:46 <marostegui> Upgrade mysql on clouddb1016 T283235 [production]
12:27 <marostegui> Upgrade mysql on clouddb1015 T283235 [production]
11:20 <jbond> upload debmonitor-client_0.3.0-1+deb10u3_all.deb to apt [production]
10:59 <topranks> Running homer for Gerrit 698162: Set up BGP peering to doh5001 in eqsin, triggering DoH /24 announcement there. [production]
09:47 <ema> pool cp1087 T278729 [production]
09:37 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host people1003.eqiad.wmnet [production]
09:33 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host people1003.eqiad.wmnet [production]
09:33 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host people2002.codfw.wmnet [production]
09:30 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host people2002.codfw.wmnet [production]
09:17 <marostegui@cumin1001> dbctl commit (dc=all): 'db1110 (re)pooling @ 100%: Repool db1110', diff saved to https://phabricator.wikimedia.org/P16304 and previous config saved to /var/cache/conftool/dbconfig/20210604-091742-root.json [production]
09:06 <ema> reboot cp1087 T278729 [production]
09:02 <marostegui@cumin1001> dbctl commit (dc=all): 'db1110 (re)pooling @ 75%: Repool db1110', diff saved to https://phabricator.wikimedia.org/P16303 and previous config saved to /var/cache/conftool/dbconfig/20210604-090239-root.json [production]
08:47 <marostegui@cumin1001> dbctl commit (dc=all): 'db1110 (re)pooling @ 50%: Repool db1110', diff saved to https://phabricator.wikimedia.org/P16302 and previous config saved to /var/cache/conftool/dbconfig/20210604-084735-root.json [production]
08:33 <marostegui> Upgrade db1110 T283235 [production]
08:32 <marostegui@cumin1001> dbctl commit (dc=all): 'db1110 (re)pooling @ 25%: Repool db1110', diff saved to https://phabricator.wikimedia.org/P16301 and previous config saved to /var/cache/conftool/dbconfig/20210604-083232-root.json [production]