2601-2650 of 10000 results (36ms)
2021-04-01 ยง
19:51 <kharlan@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . [production]
19:37 <mutante> pooled parse2001 again after twentyaftefour rebuilt the l10n cache for wmf.37 which fixed it and made Apache alert recover (T268524) [production]
19:34 <mutante> mw2379, mw2380, mw2381, mw2382 - rebooting [production]
19:34 <twentyafterfour@deploy1002> scap sync-l10n completed (1.36.0-wmf.37) (duration: 02m 38s) [production]
19:30 <mutante> depooled parse2001 because on train deployment it caused "MWException: No localisation cache found for English" and then "HTTP CRITICAL: HTTP/1.1 500 Internal Server Error" (T268524) [production]
19:29 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=parse2001.codfw.wmnet [production]
19:28 <dzahn@cumin1001> conftool action : set/pooled=inactive; selector: name=parse2001.codfw.wmnet [production]
19:27 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=parse2001.codfw.wmnet [production]
19:21 <twentyafterfour@deploy1002> rebuilt and synchronized wikiversions files: group2 wikis to 1.36.0-wmf.37 refs T278343 [production]
18:59 <mutante> creating mcrouter certs for mw2379 thorugh mw2382 [production]
18:35 <Urbanecm> Morning B&C window done [production]
18:33 <urbanecm@deploy1002> Synchronized php-1.36.0-wmf.37/extensions/WikibaseMediaInfo/resources/mediasearch-vue/components/base/Dialog.vue: e77f2b98a4fcb7d9cf74c45caeb7cfbc68a063d0: Use appendChild() instead of append() (T278448) (duration: 01m 09s) [production]
18:31 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: b485d1ca6779a03912345a094fa1101cef5f091a: Enable SandboxLink extension in ptwikinews (T278634) (duration: 01m 12s) [production]
17:49 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on bast1003.wikimedia.org with reason: REIMAGE [production]
17:47 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on bast1003.wikimedia.org with reason: REIMAGE [production]
17:24 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:21 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
16:59 <Urbanecm> Start server-side upload of two files (T279082, T279081) [production]
16:44 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: name=aqs1007.eqiad.wmnet [production]
16:39 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: a7acf3357d5d148bad11a2d2718b4da56e1a0cb8: hrwiki: Fix help panel links (T275684) (duration: 01m 10s) [production]
16:25 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2396.codfw.wmnet with reason: REIMAGE [production]
16:23 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2396.codfw.wmnet with reason: REIMAGE [production]
16:02 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2395.codfw.wmnet with reason: REIMAGE [production]
16:00 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2395.codfw.wmnet with reason: REIMAGE [production]
15:58 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2394.codfw.wmnet with reason: REIMAGE [production]
15:56 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2394.codfw.wmnet with reason: REIMAGE [production]
15:39 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2393.codfw.wmnet with reason: REIMAGE [production]
15:37 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2393.codfw.wmnet with reason: REIMAGE [production]
15:32 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2391.codfw.wmnet with reason: REIMAGE [production]
15:30 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2391.codfw.wmnet with reason: REIMAGE [production]
15:05 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2392.codfw.wmnet with reason: REIMAGE [production]
15:03 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2392.codfw.wmnet with reason: REIMAGE [production]
14:52 <volans> uploaded python3-wmflib_0.0.7 to bullseye-wikimedia [production]
14:41 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2390.codfw.wmnet with reason: REIMAGE [production]
14:39 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2390.codfw.wmnet with reason: REIMAGE [production]
14:22 <effie> disable puppet on mw* canaries, rolling depool and pooling of canaries [production]
14:06 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-worker1001.eqiad.wmnet with reason: REIMAGE [production]
14:04 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-worker1001.eqiad.wmnet with reason: REIMAGE [production]
14:01 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2389.codfw.wmnet with reason: REIMAGE [production]
13:59 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2389.codfw.wmnet with reason: REIMAGE [production]
13:53 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2388.codfw.wmnet with reason: REIMAGE [production]
13:51 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2388.codfw.wmnet with reason: REIMAGE [production]
13:24 <ema> cp3054: reboot with Linux 4.19.181+1 -- the kernel was not upgraded earlier during T273278 reboots due to broken dpkg status [production]
13:16 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1022.eqiad.wmnet [production]
13:07 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ganeti1022.eqiad.wmnet [production]
12:59 <dcaro@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
12:53 <dcaro@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
12:51 <dcaro@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
12:47 <moritzm> drain ganeti1022 [production]
12:46 <dcaro@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]