851-900 of 10000 results (21ms)
2021-03-12 §
09:43 <jayme@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
09:28 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mx1001.wikimedia.org [production]
09:25 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host mx1001.wikimedia.org [production]
09:16 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mx2001.wikimedia.org [production]
09:11 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host mx2001.wikimedia.org [production]
09:07 <zpapierski@deploy1002> Finished deploy [wikimedia/discovery/analytics@9a408b2]: T273847 export queries to relforge dag deployment - elastic-template handling (duration: 01m 35s) [production]
09:05 <zpapierski@deploy1002> Started deploy [wikimedia/discovery/analytics@9a408b2]: T273847 export queries to relforge dag deployment - elastic-template handling [production]
09:00 <zpapierski@deploy1002> Finished deploy [wikimedia/discovery/analytics@9a408b2]: T273847 export queries to relforge dag deployment - elastic-template handling (duration: 00m 09s) [production]
09:00 <zpapierski@deploy1002> Started deploy [wikimedia/discovery/analytics@9a408b2]: T273847 export queries to relforge dag deployment - elastic-template handling [production]
08:59 <zpapierski@deploy1002> Finished deploy [wikimedia/discovery/analytics@9a408b2]: T273847 export queries to relforge dag deployment - elastic-template handling (duration: 00m 10s) [production]
08:59 <zpapierski@deploy1002> Started deploy [wikimedia/discovery/analytics@9a408b2]: T273847 export queries to relforge dag deployment - elastic-template handling [production]
08:52 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pybal-test2003.codfw.wmnet [production]
08:49 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host pybal-test2003.codfw.wmnet [production]
08:47 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pybal-test2002.codfw.wmnet [production]
08:44 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host pybal-test2002.codfw.wmnet [production]
08:35 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host scandium.eqiad.wmnet [production]
08:20 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host scandium.eqiad.wmnet [production]
08:01 <moritzm> installing openjpeg2 security updates [production]
07:16 <marostegui> Stop mysql on db2108 to clone db2148 [production]
07:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2108 T276742', diff saved to https://phabricator.wikimedia.org/P14811 and previous config saved to /var/cache/conftool/dbconfig/20210312-071628-marostegui.json [production]
07:14 <marostegui@cumin1001> dbctl commit (dc=all): 'db1082 (re)pooling @ 100%: Repool db1082 after schema change', diff saved to https://phabricator.wikimedia.org/P14810 and previous config saved to /var/cache/conftool/dbconfig/20210312-071400-root.json [production]
07:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2148 T276742', diff saved to https://phabricator.wikimedia.org/P14809 and previous config saved to /var/cache/conftool/dbconfig/20210312-070219-marostegui.json [production]
06:58 <marostegui@cumin1001> dbctl commit (dc=all): 'db1082 (re)pooling @ 60%: Repool db1082 after schema change', diff saved to https://phabricator.wikimedia.org/P14808 and previous config saved to /var/cache/conftool/dbconfig/20210312-065857-root.json [production]
06:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1146:3314 for table checking T276742', diff saved to https://phabricator.wikimedia.org/P14807 and previous config saved to /var/cache/conftool/dbconfig/20210312-065008-marostegui.json [production]
06:43 <marostegui@cumin1001> dbctl commit (dc=all): 'db1082 (re)pooling @ 30%: Repool db1082 after schema change', diff saved to https://phabricator.wikimedia.org/P14806 and previous config saved to /var/cache/conftool/dbconfig/20210312-064353-root.json [production]
06:30 <marostegui> Deploy schema change on s2 codfw master, lag will appear - T276150 T276156 [production]
06:28 <marostegui@cumin1001> dbctl commit (dc=all): 'db1082 (re)pooling @ 10%: Repool db1082 after schema change', diff saved to https://phabricator.wikimedia.org/P14805 and previous config saved to /var/cache/conftool/dbconfig/20210312-062850-root.json [production]
06:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1082 for schema change', diff saved to https://phabricator.wikimedia.org/P14804 and previous config saved to /var/cache/conftool/dbconfig/20210312-061306-marostegui.json [production]
06:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db1088 from dbctl T276025', diff saved to https://phabricator.wikimedia.org/P14803 and previous config saved to /var/cache/conftool/dbconfig/20210312-061118-marostegui.json [production]
04:14 <eileen> tools revision changed from d64b2f8cee to 532f8ecb33 [production]
01:30 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw2215.codfw.wmnet [production]
00:58 <mutante> shutting down mw2215 [production]
00:57 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission for hosts mw2215.codfw.wmnet [production]
2021-03-11 §
22:55 <mutante> depooled mw2224 through mw2228 but not removing from DSH groups yet (T277119) [production]
22:55 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2228.codfw.wmnet [production]
22:55 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2227.codfw.wmnet [production]
22:55 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2226.codfw.wmnet [production]
22:54 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2225.codfw.wmnet [production]
22:54 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2224.codfw.wmnet [production]
22:50 <dzahn@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:48 <dzahn@cumin1001> START - Cookbook sre.dns.netbox [production]
22:47 <mutante> running DNS cookbook in an attempt to remove mw2216 [production]
22:47 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts mw2216.codfw.wmnet [production]
22:41 <brennen@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.34 [production]
22:36 <brennen> train status: 1.36.0-wmf.34 (T274938): T277229 and T266517 related issues hopefully resolved, rolling forward to all wikis [production]
22:34 <brennen@deploy1002> Synchronized php-1.36.0-wmf.34/extensions/WikimediaEvents/modules/ext.wikimediaEvents/clientError.js: Backport: [[gerrit:670879|Do not log script errors without file uri (T266517)]] (duration: 01m 07s) [production]
22:33 <dzahn@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:30 <brennen@deploy1002> Synchronized php-1.36.0-wmf.34/extensions/MobileFrontend/includes/: Backport: [[gerrit:670877|Revert "Fix: Save user options only once when Advanced Mode is toggled" (T277229)]] (duration: 01m 09s) [production]
22:28 <dzahn@cumin1001> START - Cookbook sre.dns.netbox [production]
21:57 <Amir1> run populate pages in cognate (T259360) [production]