3101-3150 of 10000 results (28ms)
2020-11-25 ยง
15:03 <akosiaris@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
14:56 <moritzm> installing krb5 security updates for Buster [production]
14:55 <akosiaris@cumin1001> START - Cookbook sre.ganeti.makevm [production]
14:55 <akosiaris@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
14:55 <akosiaris@cumin1001> START - Cookbook sre.ganeti.makevm [production]
14:26 <akosiaris@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:00 <akosiaris@cumin1001> START - Cookbook sre.dns.netbox [production]
13:56 <akosiaris@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
13:44 <akosiaris@cumin1001> START - Cookbook sre.dns.netbox [production]
13:43 <akosiaris> assign IPs to kubestage200{1,2,3}.codfw.wmnet, kubestagemaster2001.codfw.wmnet in netbox T268747 [production]
13:14 <marostegui> Deploy schema change on commonswiki.watchlist on s4 codfw - there will be lag on s4 codfw - T268004 [production]
13:08 <akosiaris> assign IPs to kubestage200{1,2,3}.codfw.wmnet, kubestagemaster2001.codfw.wmnet in netbox [production]
12:42 <marostegui@cumin1001> dbctl commit (dc=all): 'db1129 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P13414 and previous config saved to /var/cache/conftool/dbconfig/20201125-124202-root.json [production]
12:27 <marostegui@cumin1001> dbctl commit (dc=all): 'db1129 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P13413 and previous config saved to /var/cache/conftool/dbconfig/20201125-122659-root.json [production]
12:11 <marostegui@cumin1001> dbctl commit (dc=all): 'db1129 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P13412 and previous config saved to /var/cache/conftool/dbconfig/20201125-121155-root.json [production]
11:56 <marostegui@cumin1001> dbctl commit (dc=all): 'db1129 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P13411 and previous config saved to /var/cache/conftool/dbconfig/20201125-115652-root.json [production]
11:49 <gilles@deploy1001> Finished deploy [performance/coal@be167b2]: T268724 (duration: 00m 06s) [production]
11:48 <gilles@deploy1001> Started deploy [performance/coal@be167b2]: T268724 [production]
11:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1129 for schema change', diff saved to https://phabricator.wikimedia.org/P13408 and previous config saved to /var/cache/conftool/dbconfig/20201125-114717-marostegui.json [production]
11:27 <gilles@deploy1001> Finished deploy [performance/coal@468bc50]: T268724 (duration: 00m 06s) [production]
11:27 <gilles@deploy1001> Started deploy [performance/coal@468bc50]: T268724 [production]
11:27 <jbond42> install krb5 updates to jessie hosts [production]
10:52 <jbond42> failover idp primary to idp2001 [production]
10:51 <kormat> deployed wmfmariadbpy 0.6.1 to `C:wmfmariadbpy` [production]
10:43 <kormat> uploaded wmfmariadbpy 0.6.1 to stretch+buster apt repos [production]
10:21 <jynus> upgrade wmfbackup-check package on alert* hosts [production]
10:11 <kormat> uploaded wmfmariadbpy 0.6 to stretch+buster apt repos [production]
09:54 <moritzm> uploaded krb5 1.12.1+dfsg-19+deb8u5+wmf1 to apt.wikimedia.org [production]
09:52 <marostegui@cumin1001> dbctl commit (dc=all): 'db1076 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P13405 and previous config saved to /var/cache/conftool/dbconfig/20201125-095239-root.json [production]
09:45 <marostegui> Manually install apt-get install bsd-mailx on clouddb1015, labsdb1012 and labsdb1011 - T268725 [production]
09:37 <marostegui@cumin1001> dbctl commit (dc=all): 'db1076 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P13404 and previous config saved to /var/cache/conftool/dbconfig/20201125-093736-root.json [production]
09:22 <marostegui@cumin1001> dbctl commit (dc=all): 'db1076 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P13403 and previous config saved to /var/cache/conftool/dbconfig/20201125-092232-root.json [production]
09:07 <marostegui@cumin1001> dbctl commit (dc=all): 'db1076 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P13402 and previous config saved to /var/cache/conftool/dbconfig/20201125-090729-root.json [production]
08:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1076 for schema change', diff saved to https://phabricator.wikimedia.org/P13401 and previous config saved to /var/cache/conftool/dbconfig/20201125-085216-marostegui.json [production]
08:46 <marostegui@cumin1001> dbctl commit (dc=all): 'db1074 (re)pooling @ 100%: After cloning the new clouddb hosts', diff saved to https://phabricator.wikimedia.org/P13400 and previous config saved to /var/cache/conftool/dbconfig/20201125-084603-root.json [production]
08:43 <kormat@deploy1001> Synchronized wmf-config/db-eqiad.php: Re-enable writes to es5 T268469 (duration: 00m 59s) [production]
08:34 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:34 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:31 <marostegui@cumin1001> dbctl commit (dc=all): 'db1074 (re)pooling @ 75%: After cloning the new clouddb hosts', diff saved to https://phabricator.wikimedia.org/P13399 and previous config saved to /var/cache/conftool/dbconfig/20201125-083059-root.json [production]
08:15 <marostegui@cumin1001> dbctl commit (dc=all): 'db1074 (re)pooling @ 50%: After cloning the new clouddb hosts', diff saved to https://phabricator.wikimedia.org/P13398 and previous config saved to /var/cache/conftool/dbconfig/20201125-081556-root.json [production]
08:14 <kormat> rebooting es1024 T268469 [production]
08:08 <godog> swift eqiad-prod: add weight to ms-be106[0-3] - T268435 [production]
08:07 <kormat> stopping mariadb on es1024 T268469 [production]
08:04 <kormat@deploy1001> Synchronized wmf-config/db-eqiad.php: Disable writes to es5 T268469 (duration: 00m 58s) [production]
08:02 <marostegui> Upgrade db2108 [production]
08:02 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:02 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:00 <marostegui@cumin1001> dbctl commit (dc=all): 'db1074 (re)pooling @ 25%: After cloning the new clouddb hosts', diff saved to https://phabricator.wikimedia.org/P13397 and previous config saved to /var/cache/conftool/dbconfig/20201125-080053-root.json [production]
07:19 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1130', diff saved to https://phabricator.wikimedia.org/P13396 and previous config saved to /var/cache/conftool/dbconfig/20201125-071951-marostegui.json [production]
07:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1130 for schema change', diff saved to https://phabricator.wikimedia.org/P13395 and previous config saved to /var/cache/conftool/dbconfig/20201125-071450-marostegui.json [production]