|
2021-04-01
ยง
|
| 19:57 |
<kharlan@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 19:56 |
<razzi@deploy1002> |
Finished deploy [analytics/superset/deploy@5b8de4c]: Deployment of superset fd7c9eb71e193, released after 1.0.1 (duration: 00m 12s) |
[production] |
| 19:56 |
<razzi@deploy1002> |
Started deploy [analytics/superset/deploy@5b8de4c]: Deployment of superset fd7c9eb71e193, released after 1.0.1 |
[production] |
| 19:54 |
<kharlan@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 19:54 |
<kharlan@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 19:51 |
<kharlan@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . |
[production] |
| 19:37 |
<mutante> |
pooled parse2001 again after twentyaftefour rebuilt the l10n cache for wmf.37 which fixed it and made Apache alert recover (T268524) |
[production] |
| 19:34 |
<mutante> |
mw2379, mw2380, mw2381, mw2382 - rebooting |
[production] |
| 19:34 |
<twentyafterfour@deploy1002> |
scap sync-l10n completed (1.36.0-wmf.37) (duration: 02m 38s) |
[production] |
| 19:30 |
<mutante> |
depooled parse2001 because on train deployment it caused "MWException: No localisation cache found for English" and then "HTTP CRITICAL: HTTP/1.1 500 Internal Server Error" (T268524) |
[production] |
| 19:29 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=parse2001.codfw.wmnet |
[production] |
| 19:28 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=parse2001.codfw.wmnet |
[production] |
| 19:27 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=parse2001.codfw.wmnet |
[production] |
| 19:21 |
<twentyafterfour@deploy1002> |
rebuilt and synchronized wikiversions files: group2 wikis to 1.36.0-wmf.37 refs T278343 |
[production] |
| 18:59 |
<mutante> |
creating mcrouter certs for mw2379 thorugh mw2382 |
[production] |
| 18:35 |
<Urbanecm> |
Morning B&C window done |
[production] |
| 18:33 |
<urbanecm@deploy1002> |
Synchronized php-1.36.0-wmf.37/extensions/WikibaseMediaInfo/resources/mediasearch-vue/components/base/Dialog.vue: e77f2b98a4fcb7d9cf74c45caeb7cfbc68a063d0: Use appendChild() instead of append() (T278448) (duration: 01m 09s) |
[production] |
| 18:31 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: b485d1ca6779a03912345a094fa1101cef5f091a: Enable SandboxLink extension in ptwikinews (T278634) (duration: 01m 12s) |
[production] |
| 17:49 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on bast1003.wikimedia.org with reason: REIMAGE |
[production] |
| 17:47 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on bast1003.wikimedia.org with reason: REIMAGE |
[production] |
| 17:24 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 17:21 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:59 |
<Urbanecm> |
Start server-side upload of two files (T279082, T279081) |
[production] |
| 16:44 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=aqs1007.eqiad.wmnet |
[production] |
| 16:39 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: a7acf3357d5d148bad11a2d2718b4da56e1a0cb8: hrwiki: Fix help panel links (T275684) (duration: 01m 10s) |
[production] |
| 16:25 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2396.codfw.wmnet with reason: REIMAGE |
[production] |
| 16:23 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2396.codfw.wmnet with reason: REIMAGE |
[production] |
| 16:02 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2395.codfw.wmnet with reason: REIMAGE |
[production] |
| 16:00 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2395.codfw.wmnet with reason: REIMAGE |
[production] |
| 15:58 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2394.codfw.wmnet with reason: REIMAGE |
[production] |
| 15:56 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2394.codfw.wmnet with reason: REIMAGE |
[production] |
| 15:39 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2393.codfw.wmnet with reason: REIMAGE |
[production] |
| 15:37 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2393.codfw.wmnet with reason: REIMAGE |
[production] |
| 15:32 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2391.codfw.wmnet with reason: REIMAGE |
[production] |
| 15:30 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2391.codfw.wmnet with reason: REIMAGE |
[production] |
| 15:05 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2392.codfw.wmnet with reason: REIMAGE |
[production] |
| 15:03 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2392.codfw.wmnet with reason: REIMAGE |
[production] |
| 14:52 |
<volans> |
uploaded python3-wmflib_0.0.7 to bullseye-wikimedia |
[production] |
| 14:41 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2390.codfw.wmnet with reason: REIMAGE |
[production] |
| 14:39 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2390.codfw.wmnet with reason: REIMAGE |
[production] |
| 14:22 |
<effie> |
disable puppet on mw* canaries, rolling depool and pooling of canaries |
[production] |
| 14:06 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-worker1001.eqiad.wmnet with reason: REIMAGE |
[production] |
| 14:04 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-worker1001.eqiad.wmnet with reason: REIMAGE |
[production] |
| 14:01 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2389.codfw.wmnet with reason: REIMAGE |
[production] |
| 13:59 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2389.codfw.wmnet with reason: REIMAGE |
[production] |
| 13:53 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2388.codfw.wmnet with reason: REIMAGE |
[production] |
| 13:51 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2388.codfw.wmnet with reason: REIMAGE |
[production] |
| 13:24 |
<ema> |
cp3054: reboot with Linux 4.19.181+1 -- the kernel was not upgraded earlier during T273278 reboots due to broken dpkg status |
[production] |
| 13:16 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1022.eqiad.wmnet |
[production] |
| 13:07 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ganeti1022.eqiad.wmnet |
[production] |