351-400 of 10000 results (55ms)
2019-11-19 ยง
13:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1092 for schema change', diff saved to https://phabricator.wikimedia.org/P9673 and previous config saved to /var/cache/conftool/dbconfig/20191119-133850-marostegui.json [production]
13:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1101:3318 after schema change', diff saved to https://phabricator.wikimedia.org/P9672 and previous config saved to /var/cache/conftool/dbconfig/20191119-133704-marostegui.json [production]
13:34 <ema@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
13:33 <ema@cumin2001> START - Cookbook sre.hosts.downtime [production]
13:14 <ema> depool cp2001 and reimage as text_ats T227432 [production]
12:42 <jbond42> add libapache2-mod-auth-cas 1.2-1 to stretch-wikimedia repo [production]
12:28 <effie> enable puppet on P:mediawiki::php and *.eqiad.wmnet [production]
12:22 <effie> enable puppet on P:mediawiki::php and *.codfw.wmnet [production]
12:12 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db1067 from config T238297 (duration: 00m 52s) [production]
12:11 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db1067 from config T238297 (duration: 00m 52s) [production]
11:41 <gehel> depooling wdqs1004 - T231411 [production]
11:37 <gehel> restarting wdqs blazegraph on wdqs1004 - T231411 [production]
11:29 <marostegui> Upgrade dbstore1003 (3311,3315,3317) [production]
11:16 <gehel> restarting wdqs updater on wdqs1004 - T231411 [production]
10:36 <marostegui> Compress and upgrade db1098:3316 [production]
10:35 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1096:3316 for upgrade and compression', diff saved to https://phabricator.wikimedia.org/P9671 and previous config saved to /var/cache/conftool/dbconfig/20191119-103540-marostegui.json [production]
10:34 <marostegui> Compress and upgrade db1098:3317 [production]
10:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1098:3317 for upgrade and compression', diff saved to https://phabricator.wikimedia.org/P9670 and previous config saved to /var/cache/conftool/dbconfig/20191119-103426-marostegui.json [production]
10:29 <marostegui> Upgrade db2077 [production]
10:24 <marostegui> Upgrade db2120 db2121 db2122 [production]
10:10 <marostegui> Upgrade MySQL on db2086 db2087 db2100 [production]
10:06 <godog> repool centrallog2001 [production]
09:40 <effie> disable puppet on P:mediawiki::php - T229792 [production]
09:21 <moritzm> installing ncurses security updates [production]
09:20 <moritzm> rolling restart of nginx on acmechief/puppetdb to pick up libxslt security updates [production]
09:08 <moritzm> installing libxslt security updates [production]
09:08 <marostegui> Deploy schema change on db1101:3318 [production]
09:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1101:3318 for schema change', diff saved to https://phabricator.wikimedia.org/P9669 and previous config saved to /var/cache/conftool/dbconfig/20191119-090823-marostegui.json [production]
09:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1099:3318 after schema change', diff saved to https://phabricator.wikimedia.org/P9668 and previous config saved to /var/cache/conftool/dbconfig/20191119-090745-marostegui.json [production]
09:05 <marostegui> Repool labsbdb1010 [production]
07:33 <mobrovac@deploy1001> Synchronized wmf-config/CommonSettings-labs.php: Enable math links in Beta - T208758 (duration: 00m 53s) [production]
06:45 <marostegui> Stop MySQL on db2061 T238526 [production]
06:44 <marostegui> Remove db2061 from tendril and zarcillo T238526 [production]
06:39 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db2061 from config T238526 (duration: 00m 52s) [production]
06:38 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db2061 from config T238526 (duration: 00m 53s) [production]
06:26 <vgutierrez> Move cp1089 from nginx to ats-tls - T231627 [production]
06:20 <marostegui> Depool labsdb1010 for upgrade [production]
06:02 <marostegui@cumin2001> dbctl commit (dc=all): 'Promote db1131 to s6 master and remove read-only from s6 T235469', diff saved to https://phabricator.wikimedia.org/P9667 and previous config saved to /var/cache/conftool/dbconfig/20191119-060203-marostegui.json [production]
06:01 <marostegui@cumin2001> dbctl commit (dc=all): 'Set s6 as read-only for maintenance T235469', diff saved to https://phabricator.wikimedia.org/P9666 and previous config saved to /var/cache/conftool/dbconfig/20191119-060122-marostegui.json [production]
06:01 <marostegui> Starting s6 failover from db1061 to db1131 - T235469 [production]
05:37 <eileen> process control - I reverted the above to check some stuff first [production]
05:36 <vgutierrez> Move cp1087 from nginx to ats-tls - T231627 [production]
05:26 <marostegui> Deploy schema change on db1099:3318 [production]
05:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1099:3318 for schema change', diff saved to https://phabricator.wikimedia.org/P9665 and previous config saved to /var/cache/conftool/dbconfig/20191119-052632-marostegui.json [production]
05:25 <marostegui> Compress db1097:3314 [production]
05:24 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1097:3314 for compression', diff saved to https://phabricator.wikimedia.org/P9664 and previous config saved to /var/cache/conftool/dbconfig/20191119-052412-marostegui.json [production]
05:17 <vgutierrez> Move cp1085 from nginx to ats-tls - T231627 [production]
05:14 <marostegui> Compress tables on db1105:3311 [production]
05:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1105:3311 for compression', diff saved to https://phabricator.wikimedia.org/P9663 and previous config saved to /var/cache/conftool/dbconfig/20191119-051344-marostegui.json [production]
05:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1105:3312 after compression', diff saved to https://phabricator.wikimedia.org/P9662 and previous config saved to /var/cache/conftool/dbconfig/20191119-051259-marostegui.json [production]