1901-1950 of 10000 results (39ms)
2022-01-21 §
05:57 <marostegui@cumin1001> dbctl commit (dc=all): 'es1022 (re)pooling @ 5%: repooling after on-site maintenance', diff saved to https://phabricator.wikimedia.org/P18951 and previous config saved to /var/cache/conftool/dbconfig/20220121-055732-root.json [production]
05:49 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host es2030.codfw.wmnet with OS bullseye [production]
05:42 <marostegui@cumin1001> dbctl commit (dc=all): 'es1022 (re)pooling @ 1%: repooling after on-site maintenance', diff saved to https://phabricator.wikimedia.org/P18950 and previous config saved to /var/cache/conftool/dbconfig/20220121-054228-root.json [production]
2022-01-20 §
22:40 <inflatador> running puppet-merge for https://gerrit.wikimedia.org/r/755810 [production]
22:38 <inflatador> running puppet-merge for ^^ [production]
22:27 <urandom> rolling restart of Cassandra, aqs-next -- T298516 [production]
21:04 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host backup1008.eqiad.wmnet with OS buster [production]
20:58 <jhathaway> rebotting mx1001 to test new kernel [production]
20:40 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
20:38 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
20:38 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
20:37 <urandom> upgrading Cassandra to 3.11.11, aqs1010 -- T298516 [production]
20:37 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
20:36 <jhuneidi@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.38.0-wmf.18 refs T293959 [production]
20:34 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host backup1008.eqiad.wmnet with OS buster [production]
20:32 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
20:31 <jhuneidi@deploy1002> Synchronized php-1.38.0-wmf.18/extensions/DiscussionTools/includes/HeadingItem.php: Backport: [[gerrit:755684|Prevent assertion failure caused by empty headings (T299583)]] (duration: 00m 50s) [production]
20:31 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
20:31 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
20:30 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
19:44 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
19:43 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
19:43 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
19:42 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
19:38 <bd808@deploy1002> Synchronized wmf-config/wikitech.php: wikitech: Remove password clear on block (duration: 00m 50s) [production]
19:19 <jhathaway> rebooting mx1001 to test new kernel [production]
19:17 <dzahn@deploy1002> helmfile [eqiad] DONE helmfile.d/services/miscweb: sync on main [production]
19:16 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
19:15 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
19:15 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
19:14 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
19:14 <dzahn@deploy1002> helmfile [eqiad] START helmfile.d/services/miscweb: apply on main [production]
19:13 <dzahn@deploy1002> helmfile [codfw] DONE helmfile.d/services/miscweb: sync on main [production]
19:11 <cjming> end of UTC evening backport & config window [production]
19:10 <dzahn@deploy1002> helmfile [codfw] START helmfile.d/services/miscweb: apply on main [production]
19:10 <dzahn@deploy1002> helmfile [staging] DONE helmfile.d/services/miscweb: sync on main [production]
19:09 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
19:08 <dzahn@deploy1002> helmfile [staging] START helmfile.d/services/miscweb: apply on main [production]
19:08 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
19:08 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
19:07 <cjming@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:755745|Disable language alert for pilot wikis except thwiki, viwiki. (T295555)]] (duration: 00m 51s) [production]
19:06 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
18:41 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
18:40 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
18:40 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
18:38 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
18:33 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
18:32 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
18:32 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
18:31 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]