3401-3450 of 10000 results (42ms)
2021-11-08 §
12:38 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
12:28 <vgutierrez@cumin1001> START - Cookbook sre.hosts.reimage for host cp4026.ulsfo.wmnet with OS buster [production]
12:23 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
12:19 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:699692|Update autonyms in wmgExtraLanguageNames (T284870)]] (duration: 00m 56s) [production]
12:19 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
12:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Adjust weights for s5 codfw replicas after removing special groups from them T263127', diff saved to https://phabricator.wikimedia.org/P17708 and previous config saved to /var/cache/conftool/dbconfig/20211108-120203-marostegui.json [production]
11:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove contributions logpager recentchanges recentchangeslinked watchlist from s5 codfw T263127', diff saved to https://phabricator.wikimedia.org/P17707 and previous config saved to /var/cache/conftool/dbconfig/20211108-115945-marostegui.json [production]
11:41 <mmandere@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti6002.drmrs.wmnet with OS buster [production]
11:32 <vgutierrez@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4026.ulsfo.wmnet with OS buster [production]
11:01 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host ganeti6002.drmrs.wmnet with OS buster [production]
10:53 <hnowlan@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
10:53 <hnowlan@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
10:49 <hnowlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
10:49 <hnowlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
10:49 <vgutierrez@cumin1001> START - Cookbook sre.hosts.reimage for host cp4026.ulsfo.wmnet with OS buster [production]
10:27 <vgutierrez> depool cp4026 to be reimaged as a haproxy-tls test node - T290005 [production]
10:27 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2003.codfw.wmnet [production]
10:17 <Lucas_WMDE> Deployed patch for T294693 [production]
09:47 <XioNoX> all core routers: add drmrs to prefix lists + confed [production]
09:47 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti-test2003.codfw.wmnet [production]
09:46 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2002.codfw.wmnet [production]
09:23 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
09:22 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
09:22 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
09:21 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti-test2002.codfw.wmnet [production]
09:21 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2001.codfw.wmnet [production]
09:16 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet [production]
08:51 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
08:51 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
08:24 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality' for release 'main' . [production]
05:52 <rzl> rebooted wikitech-static via rackspace web UI - T295266 [production]
2021-11-06 §
01:43 <dduvall@deploy1002> Synchronized php-1.38.0-wmf.7/includes/parser/ParserOutput.php: Backport: [[gerrit:737079|Regression fix: do language conversion on ToC in ParserOutput::getText() (T295187)]] (duration: 00m 56s) [production]
01:42 <dduvall> emergency backport https://gerrit.wikimedia.org/r/c/mediawiki/core/+/737079 deployed and verified on mwdebug1002. syncing to all targets [production]
01:40 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
01:37 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
01:33 <dduvall> performing emergency backport deployment of https://gerrit.wikimedia.org/r/c/mediawiki/core/+/737079 [production]
2021-11-05 §
23:26 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
23:19 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
23:05 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
22:58 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
22:48 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
22:45 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
22:35 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
22:32 <dduvall> re-rolling 1.38.0-wmf.7 to all wikis due to a better of two evil regressions UBN T295187 (refs T293948) [production]
22:32 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.38.0-wmf.7 refs T293948 [production]
22:31 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
22:21 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: Revert "group0/group1 to 1.38.0-wmf.7 refs T293948" [production]
22:19 <dduvall> rolling back 1.38.0-wmf.7 from group1 and group0 due to UBN T295187 (refs T293948) [production]
20:17 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: Revert "all wikis to 1.38.0-wmf.7 refs T293948" [production]
20:09 <dduvall> rolling back 1.38.0-wmf.7 from all wikis due to UBN T295187 (refs T293948) [production]