3151-3200 of 10000 results (42ms)
2021-11-23 §
00:35 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
00:30 <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.9/extensions/WikimediaEvents: 3f860c7: fa9fbf1: WikimediaEvents bbackports (2/2; T294777) (duration: 00m 55s) [production]
00:28 <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.9/extensions/WikimediaEvents/extension.json: 3f860c72bca817c40486b90f0d8e0ffca72b2690: Restore ReadingDepth instrument (1/2) (duration: 00m 56s) [production]
00:25 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
00:21 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
00:20 <jeena> Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/739908 [production]
2021-11-22 §
23:55 <mutante> acmechief1001, acmechief-test1001: sudo systemctl restart reload-acme-chief-backend.timer [production]
23:54 <mutante> acmechief1001, acmechief-test1001: sudo systemctl start reload-acme-chief-backend.timer [production]
23:48 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-fe2011.codfw.wmnet with OS stretch [production]
23:22 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-fe2010.codfw.wmnet with OS stretch [production]
23:20 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host ms-fe2011.codfw.wmnet with OS stretch [production]
23:18 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2009.codfw.wmnet with OS stretch [production]
22:53 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host ms-fe2010.codfw.wmnet with OS stretch [production]
22:44 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host ms-fe2009.codfw.wmnet with OS stretch [production]
22:20 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2028.codfw.wmnet with OS buster [production]
21:51 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host ganeti2028.codfw.wmnet with OS buster [production]
21:42 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2027.codfw.wmnet with OS buster [production]
21:07 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host ganeti2027.codfw.wmnet with OS buster [production]
20:43 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
20:39 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
20:28 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
20:24 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
20:23 <legoktm@deploy1002> Synchronized wmf-config/PoolCounterSettings.php: Lower CirrusSearch maxqueues to be closer to number of workers (duration: 00m 56s) [production]
20:01 <ryankemper@cumin1001> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) restart with plugin upgrade (3 nodes at a time) for ElasticSearch cluster search_codfw: codfw plugin upgrade + restart - ryankemper@cumin1001 - T295705 [production]
19:49 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation restart with plugin upgrade (3 nodes at a time) for ElasticSearch cluster search_codfw: codfw plugin upgrade + restart - ryankemper@cumin1001 - T295705 [production]
19:49 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:46 <urbanecm> Evening B&C window completed [production]
19:45 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:44 <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.9/extensions/ProofreadPage/: 10b8440069ac71434274462c545c6b2b2c9182d9: Use the WikiEditor ready hook instead of using() the lib (T296033) (duration: 00m 56s) [production]
19:34 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:30 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:24 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: b6b05e30b3c9b4007fd31ab0698507d7a48d1caf: kswiki: set wgTranslateNumerals to false (T296055) (duration: 00m 55s) [production]
19:20 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:18 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 4aa8d5bf465bfc3fee2ec547718af0c779f88ef4: Enable SandboxLink on lawiki (T296073) (duration: 00m 56s) [production]
19:16 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 1c082bec4c74c156b26af4349488835902c5bacd: Enable mapframe on the Indonesian Wikipedia (T295571) (duration: 00m 56s) [production]
19:15 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:11 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:05 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
19:01 <vgutierrez> pool cp4032 (text) using HAProxy as TLS terminator - T290005 [production]
18:20 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:14 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
18:04 <ryankemper@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) restart with plugin upgrade (3 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic plugin upgrade + restart - ryankemper@cumin1001 [production]
17:50 <ryankemper@cumin1001> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) restart with plugin upgrade (3 nodes at a time) for ElasticSearch cluster search_codfw: codfw plugin upgrade + restart - ryankemper@cumin1001 - T295705 [production]
17:48 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:48 <XioNoX> repool codfw [production]
17:46 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4032.ulsfo.wmnet with OS buster [production]
17:46 <ejegg> updated fundraising python tools from d90f4c91 -> d1d7b100 [production]
17:43 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
17:32 <ebernhardson> restart both elasticsearch instances on elastic2044, reporting `connection refused` (after a brief period of `no route to host`) to masters even though the connection works outside elastic [production]
17:01 <ryankemper> T295705 Beginning rolling restart w/ plugin upgrade of `cloudelastic`: `ryankemper@cumin1001:~$ sudo cookbook sre.elasticsearch.rolling-operation cloudelastic "cloudelastic plugin upgrade + restart" --upgrade --nodes-per-run 3 --start-datetime 2021-11-22T16:59:38 --task-id T295705` on tmux `rolling_restarts_cloudelastic` [production]