|
2021-04-01
ยง
|
| 19:59 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2382.codfw.wmnet |
[production] |
| 19:59 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2381.codfw.wmnet |
[production] |
| 19:59 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2380.codfw.wmnet |
[production] |
| 19:59 |
<razzi@deploy1002> |
Finished deploy [analytics/superset/deploy@5b8de4c]: Deployment of superset fd7c9eb71e193, released after 1.0.1 (duration: 00m 21s) |
[production] |
| 19:59 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2379.codfw.wmnet |
[production] |
| 19:58 |
<razzi@deploy1002> |
Started deploy [analytics/superset/deploy@5b8de4c]: Deployment of superset fd7c9eb71e193, released after 1.0.1 |
[production] |
| 19:57 |
<kharlan@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 19:57 |
<kharlan@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 19:56 |
<razzi@deploy1002> |
Finished deploy [analytics/superset/deploy@5b8de4c]: Deployment of superset fd7c9eb71e193, released after 1.0.1 (duration: 00m 12s) |
[production] |
| 19:56 |
<razzi@deploy1002> |
Started deploy [analytics/superset/deploy@5b8de4c]: Deployment of superset fd7c9eb71e193, released after 1.0.1 |
[production] |
| 19:54 |
<kharlan@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 19:54 |
<kharlan@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 19:54 |
<razzi> |
dump superset production to an-coord1001.eqiad.wmnet:/home/razzi/superset_production_1617306805.sql just in case |
[analytics] |
| 19:51 |
<kharlan@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . |
[production] |
| 19:37 |
<mutante> |
pooled parse2001 again after twentyaftefour rebuilt the l10n cache for wmf.37 which fixed it and made Apache alert recover (T268524) |
[production] |
| 19:34 |
<mutante> |
mw2379, mw2380, mw2381, mw2382 - rebooting |
[production] |
| 19:34 |
<twentyafterfour@deploy1002> |
scap sync-l10n completed (1.36.0-wmf.37) (duration: 02m 38s) |
[production] |
| 19:30 |
<mutante> |
depooled parse2001 because on train deployment it caused "MWException: No localisation cache found for English" and then "HTTP CRITICAL: HTTP/1.1 500 Internal Server Error" (T268524) |
[production] |
| 19:29 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=parse2001.codfw.wmnet |
[production] |
| 19:28 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=parse2001.codfw.wmnet |
[production] |
| 19:27 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=parse2001.codfw.wmnet |
[production] |
| 19:21 |
<twentyafterfour@deploy1002> |
rebuilt and synchronized wikiversions files: group2 wikis to 1.36.0-wmf.37 refs T278343 |
[production] |
| 18:59 |
<mutante> |
creating mcrouter certs for mw2379 thorugh mw2382 |
[production] |
| 18:35 |
<Urbanecm> |
Morning B&C window done |
[production] |
| 18:33 |
<urbanecm@deploy1002> |
Synchronized php-1.36.0-wmf.37/extensions/WikibaseMediaInfo/resources/mediasearch-vue/components/base/Dialog.vue: e77f2b98a4fcb7d9cf74c45caeb7cfbc68a063d0: Use appendChild() instead of append() (T278448) (duration: 01m 09s) |
[production] |
| 18:31 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: b485d1ca6779a03912345a094fa1101cef5f091a: Enable SandboxLink extension in ptwikinews (T278634) (duration: 01m 12s) |
[production] |
| 18:04 |
<wm-bot> |
<lucaswerkmeister> deployed f2b128273d (l10n updates) |
[tools.lexeme-forms] |
| 17:49 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on bast1003.wikimedia.org with reason: REIMAGE |
[production] |
| 17:47 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on bast1003.wikimedia.org with reason: REIMAGE |
[production] |
| 17:24 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 17:21 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:59 |
<Urbanecm> |
Start server-side upload of two files (T279082, T279081) |
[production] |
| 16:50 |
<razzi> |
rebalance kafka partitions for webrequest_text partitions 7 and 8 |
[analytics] |
| 16:44 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=aqs1007.eqiad.wmnet |
[production] |
| 16:39 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: a7acf3357d5d148bad11a2d2718b4da56e1a0cb8: hrwiki: Fix help panel links (T275684) (duration: 01m 10s) |
[production] |
| 16:25 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2396.codfw.wmnet with reason: REIMAGE |
[production] |
| 16:23 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2396.codfw.wmnet with reason: REIMAGE |
[production] |
| 16:16 |
<Majavah> |
hard reboot unresponsive deployment-cache-text06 |
[releng] |
| 16:02 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2395.codfw.wmnet with reason: REIMAGE |
[production] |
| 16:00 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2395.codfw.wmnet with reason: REIMAGE |
[production] |
| 15:58 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2394.codfw.wmnet with reason: REIMAGE |
[production] |
| 15:56 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2394.codfw.wmnet with reason: REIMAGE |
[production] |
| 15:53 |
<dcaro> |
Removed etcd member tools-k8s-etcd-5.tools.eqiad.wmflabs, adding a new member (T267082) |
[tools] |
| 15:43 |
<dcaro> |
Removing etcd member tools-k8s-etcd-5.tools.eqiad.wmflabs (T267082) |
[tools] |
| 15:39 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2393.codfw.wmnet with reason: REIMAGE |
[production] |
| 15:37 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2393.codfw.wmnet with reason: REIMAGE |
[production] |
| 15:36 |
<dcaro> |
Added new etcd member tools-k8s-etcd-9.tools.eqiad1.wikimedia.cloud (T267082) |
[tools] |
| 15:32 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2391.codfw.wmnet with reason: REIMAGE |
[production] |
| 15:30 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2391.codfw.wmnet with reason: REIMAGE |
[production] |
| 15:18 |
<dcaro> |
adding new etcd member using the cookbook wmcs.toolforge.add_etcd_node (T267082) |
[tools] |