|
2020-06-23
§
|
| 10:55 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 10:42 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:42 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:40 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:38 |
<moritzm> |
temporarily shutdown xhgui1001/releases1002 to reshuffle Ganeti instances for reboots |
[production] |
| 10:38 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:35 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:35 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:22 |
<kormat> |
reimaging db1088 to buster T250666 |
[production] |
| 10:03 |
<jynus@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:01 |
<jynus@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:48 |
<jbond42> |
add new CI check for cloud yaml data https://gerrit.wikimedia.org/r/c/operations/puppet/+/606444/ |
[production] |
| 09:46 |
<jynus> |
stopping and reimaging db2101 into buster T254871 |
[production] |
| 09:32 |
<marostegui> |
Reload haproxy on dbproxy1012 and dbproxy1014 to test db1097 as secondary for 24h T254556 |
[production] |
| 08:46 |
<ema> |
mwmaint1002: add uid=abban,ou=people,dc=wikimedia,dc=org to group 'nda' T255775 |
[production] |
| 08:38 |
<XioNoX> |
re-enable peering BGP sessions on AMS-IX - T253970 |
[production] |
| 08:03 |
<moritzm> |
draining ganeti1007 for eventual reboot |
[production] |
| 07:58 |
<XioNoX> |
restart scs-a8-eqiad - T256101 |
[production] |
| 07:51 |
<marostegui@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:49 |
<marostegui@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:42 |
<marostegui> |
Deploy schema change on db1088 |
[production] |
| 07:30 |
<marostegui> |
Reimage db2133 (m2 codfw master) to Buster (this will trigger haproxy IRC alert) T250666 |
[production] |
| 07:01 |
<marostegui@cumin2001> |
dbctl commit (dc=all): 'Fully repool db1118', diff saved to https://phabricator.wikimedia.org/P11637 and previous config saved to /var/cache/conftool/dbconfig/20200623-070120-marostegui.json |
[production] |
| 06:06 |
<XioNoX> |
disable peering BGP sessions on AMS-IX - T253970 |
[production] |
| 05:24 |
<marostegui> |
Compress InnoDB on db1080 T254462 |
[production] |
| 05:23 |
<marostegui@cumin2001> |
dbctl commit (dc=all): 'Depool db1080 for InnoDB compression', diff saved to https://phabricator.wikimedia.org/P11636 and previous config saved to /var/cache/conftool/dbconfig/20200623-052350-marostegui.json |
[production] |
| 05:22 |
<marostegui@cumin2001> |
dbctl commit (dc=all): 'Slowly repool db1118', diff saved to https://phabricator.wikimedia.org/P11635 and previous config saved to /var/cache/conftool/dbconfig/20200623-052254-marostegui.json |
[production] |
| 05:12 |
<marostegui@cumin2001> |
dbctl commit (dc=all): 'Slowly repool db1118', diff saved to https://phabricator.wikimedia.org/P11634 and previous config saved to /var/cache/conftool/dbconfig/20200623-051159-marostegui.json |
[production] |
| 05:03 |
<marostegui@cumin2001> |
dbctl commit (dc=all): 'Slowly repool db1118', diff saved to https://phabricator.wikimedia.org/P11633 and previous config saved to /var/cache/conftool/dbconfig/20200623-050314-marostegui.json |
[production] |
|
2020-06-22
§
|
| 23:40 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: touch for T247330 (duration: 00m 56s) |
[production] |
| 23:36 |
<catrope@deploy1001> |
Synchronized dblists/: Close trwikinews (T247330) (duration: 00m 58s) |
[production] |
| 23:28 |
<RoanKattouw> |
Synchronized wmf-config/InitialiseSettings.php: Create rollbacker group on elwiktionary (T255569) (typoed the task number before) |
[production] |
| 23:26 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Create rollbacker group on elwiktionary (T225569) (duration: 00m 56s) |
[production] |
| 23:21 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Add localized sitename for bewikibooks (T253962) (duration: 00m 57s) |
[production] |
| 23:16 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Add domains to wgCopyUploadsDomains (T255336, T255363, T255386, T255313) (duration: 01m 01s) |
[production] |
| 22:39 |
<bstorm_> |
downtimed labstore1005 to prevent an alert during puppet merge T253353 |
[production] |
| 22:38 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 22:35 |
<volans@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 22:16 |
<ebernhardson@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@f2002c8]: bump glent jar to 0.2.2 (duration: 00m 56s) |
[production] |
| 22:15 |
<ebernhardson@deploy1001> |
Started deploy [wikimedia/discovery/analytics@f2002c8]: bump glent jar to 0.2.2 |
[production] |
| 22:12 |
<volans> |
cleanup interfaces and addresses in Netbox for offline servers - T233183 |
[production] |
| 21:59 |
<ebernhardson@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@6e7f9f7]: bump glent jar to 0.2.2 (duration: 00m 18s) |
[production] |
| 21:58 |
<ebernhardson@deploy1001> |
Started deploy [wikimedia/discovery/analytics@6e7f9f7]: bump glent jar to 0.2.2 |
[production] |
| 17:19 |
<mutante> |
gerrit1002 - let puppet remove [database] secttion from config; restart gerrit another time |
[production] |
| 17:14 |
<mutante> |
gerrit1002 (gerrit-test): re-enabled puppet, restarted gerrit service |
[production] |
| 16:58 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 16:49 |
<volans@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 15:05 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 15:01 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 14:48 |
<moritzm> |
installing mutt security updates |
[production] |