|
2020-06-23
§
|
| 08:03 |
<moritzm> |
draining ganeti1007 for eventual reboot |
[production] |
| 07:58 |
<XioNoX> |
restart scs-a8-eqiad - T256101 |
[production] |
| 07:51 |
<marostegui@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:49 |
<marostegui@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:42 |
<marostegui> |
Deploy schema change on db1088 |
[production] |
| 07:30 |
<marostegui> |
Reimage db2133 (m2 codfw master) to Buster (this will trigger haproxy IRC alert) T250666 |
[production] |
| 07:01 |
<marostegui@cumin2001> |
dbctl commit (dc=all): 'Fully repool db1118', diff saved to https://phabricator.wikimedia.org/P11637 and previous config saved to /var/cache/conftool/dbconfig/20200623-070120-marostegui.json |
[production] |
| 06:06 |
<XioNoX> |
disable peering BGP sessions on AMS-IX - T253970 |
[production] |
| 05:24 |
<marostegui> |
Compress InnoDB on db1080 T254462 |
[production] |
| 05:23 |
<marostegui@cumin2001> |
dbctl commit (dc=all): 'Depool db1080 for InnoDB compression', diff saved to https://phabricator.wikimedia.org/P11636 and previous config saved to /var/cache/conftool/dbconfig/20200623-052350-marostegui.json |
[production] |
| 05:22 |
<marostegui@cumin2001> |
dbctl commit (dc=all): 'Slowly repool db1118', diff saved to https://phabricator.wikimedia.org/P11635 and previous config saved to /var/cache/conftool/dbconfig/20200623-052254-marostegui.json |
[production] |
| 05:12 |
<marostegui@cumin2001> |
dbctl commit (dc=all): 'Slowly repool db1118', diff saved to https://phabricator.wikimedia.org/P11634 and previous config saved to /var/cache/conftool/dbconfig/20200623-051159-marostegui.json |
[production] |
| 05:03 |
<marostegui@cumin2001> |
dbctl commit (dc=all): 'Slowly repool db1118', diff saved to https://phabricator.wikimedia.org/P11633 and previous config saved to /var/cache/conftool/dbconfig/20200623-050314-marostegui.json |
[production] |
|
2020-06-22
§
|
| 23:40 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: touch for T247330 (duration: 00m 56s) |
[production] |
| 23:36 |
<catrope@deploy1001> |
Synchronized dblists/: Close trwikinews (T247330) (duration: 00m 58s) |
[production] |
| 23:28 |
<RoanKattouw> |
Synchronized wmf-config/InitialiseSettings.php: Create rollbacker group on elwiktionary (T255569) (typoed the task number before) |
[production] |
| 23:26 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Create rollbacker group on elwiktionary (T225569) (duration: 00m 56s) |
[production] |
| 23:21 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Add localized sitename for bewikibooks (T253962) (duration: 00m 57s) |
[production] |
| 23:16 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Add domains to wgCopyUploadsDomains (T255336, T255363, T255386, T255313) (duration: 01m 01s) |
[production] |
| 22:39 |
<bstorm_> |
downtimed labstore1005 to prevent an alert during puppet merge T253353 |
[production] |
| 22:38 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 22:35 |
<volans@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 22:16 |
<ebernhardson@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@f2002c8]: bump glent jar to 0.2.2 (duration: 00m 56s) |
[production] |
| 22:15 |
<ebernhardson@deploy1001> |
Started deploy [wikimedia/discovery/analytics@f2002c8]: bump glent jar to 0.2.2 |
[production] |
| 22:12 |
<volans> |
cleanup interfaces and addresses in Netbox for offline servers - T233183 |
[production] |
| 21:59 |
<ebernhardson@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@6e7f9f7]: bump glent jar to 0.2.2 (duration: 00m 18s) |
[production] |
| 21:58 |
<ebernhardson@deploy1001> |
Started deploy [wikimedia/discovery/analytics@6e7f9f7]: bump glent jar to 0.2.2 |
[production] |
| 17:19 |
<mutante> |
gerrit1002 - let puppet remove [database] secttion from config; restart gerrit another time |
[production] |
| 17:14 |
<mutante> |
gerrit1002 (gerrit-test): re-enabled puppet, restarted gerrit service |
[production] |
| 16:58 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 16:49 |
<volans@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 15:05 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 15:01 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 14:48 |
<moritzm> |
installing mutt security updates |
[production] |
| 14:47 |
<Amir1> |
creating shnwiktionary is done |
[production] |
| 14:44 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 02m 58s) |
[production] |
| 14:42 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 14:41 |
<ladsgroup@deploy1001> |
Synchronized static/images/project-logos/: Creating shnwiktionary (T253029) (duration: 00m 56s) |
[production] |
| 14:40 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Creating shnwiktionary (T253029) (duration: 00m 56s) |
[production] |
| 14:39 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 14:37 |
<ladsgroup@deploy1001> |
rebuilt and synchronized wikiversions files: Creating shnwiktionary (T253029) |
[production] |
| 14:36 |
<ladsgroup@deploy1001> |
Synchronized dblists: Creating shnwiktionary (T253029) (duration: 00m 58s) |
[production] |
| 14:16 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 14:10 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 14:05 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 13:59 |
<moritzm> |
re-enabling Puppet in codfw |
[production] |
| 13:55 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 13:51 |
<moritzm> |
disable Puppet in codfw to reduce puppetdb2002 memory activity, unblocking the migration of the Ganeti instance for a reboot |
[production] |
| 13:19 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Bump eventlogging_Test schema version to 1.1.0 to pick up client_dt and set wgEventLoggingServiceUri for all wikis - T238230 (duration: 00m 58s) |
[production] |
| 13:11 |
<marostegui> |
Stop MySQL on db2078 instances |
[production] |