|
2020-06-22
ยง
|
| 13:59 |
<moritzm> |
re-enabling Puppet in codfw |
[production] |
| 13:55 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 13:51 |
<moritzm> |
disable Puppet in codfw to reduce puppetdb2002 memory activity, unblocking the migration of the Ganeti instance for a reboot |
[production] |
| 13:19 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Bump eventlogging_Test schema version to 1.1.0 to pick up client_dt and set wgEventLoggingServiceUri for all wikis - T238230 (duration: 00m 58s) |
[production] |
| 13:11 |
<marostegui> |
Stop MySQL on db2078 instances |
[production] |
| 12:53 |
<vgutierrez> |
upgrade to trafficserver 8.0.8~rc0-1wm1 on cp5006 and cp5012 |
[production] |
| 12:45 |
<moritzm> |
draining ganeti2007 for eventual reboot |
[production] |
| 12:42 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 12:38 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 12:31 |
<akosiaris> |
failover logstash2023 from ganeti2007->ganeti2023 for migration_downtime change to apply |
[production] |
| 12:26 |
<volans@deploy1001> |
Finished deploy [homer/deploy@e9acec8]: Release v0.2.3 on cumin1001 now on buster (duration: 01m 25s) |
[production] |
| 12:24 |
<volans@deploy1001> |
Started deploy [homer/deploy@e9acec8]: Release v0.2.3 on cumin1001 now on buster |
[production] |
| 12:22 |
<volans@deploy1001> |
Finished deploy [homer/deploy@e9acec8]: Release v0.2.3 on cumin1001 now on buster (duration: 00m 03s) |
[production] |
| 12:22 |
<volans@deploy1001> |
Started deploy [homer/deploy@e9acec8]: Release v0.2.3 on cumin1001 now on buster |
[production] |
| 11:53 |
<Urbanecm> |
EU B&C window done |
[production] |
| 11:50 |
<urbanecm@deploy1001> |
Synchronized php-1.35.0-wmf.37/extensions/VisualEditor/modules/: Backport: 0a08066: Revert "Allow generic params to be passed to getWikitextFragment" (T255785) (duration: 00m 58s) |
[production] |
| 11:45 |
<marostegui@cumin2001> |
dbctl commit (dc=all): 'Fully repool db1094', diff saved to https://phabricator.wikimedia.org/P11627 and previous config saved to /var/cache/conftool/dbconfig/20200622-114554-marostegui.json |
[production] |
| 11:40 |
<moritzm> |
draining ganeti2008 for eventual reboot |
[production] |
| 11:37 |
<volans@deploy1001> |
Finished deploy [homer/deploy@e9acec8]: Release v0.2.3 on cumin1001 now on buster (duration: 00m 28s) |
[production] |
| 11:37 |
<volans@deploy1001> |
Started deploy [homer/deploy@e9acec8]: Release v0.2.3 on cumin1001 now on buster |
[production] |
| 11:34 |
<marostegui@cumin2001> |
dbctl commit (dc=all): 'Slowly repool db1094', diff saved to https://phabricator.wikimedia.org/P11625 and previous config saved to /var/cache/conftool/dbconfig/20200622-113401-marostegui.json |
[production] |
| 11:30 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 74e8295: IS: Cleanup some redundant rows (duration: 00m 56s) |
[production] |
| 11:29 |
<Urbanecm> |
Run namespaceDupes.php for zh* projects (T165593) |
[production] |
| 11:24 |
<marostegui@cumin2001> |
dbctl commit (dc=all): 'Slowly repool db1094', diff saved to https://phabricator.wikimedia.org/P11623 and previous config saved to /var/cache/conftool/dbconfig/20200622-112451-marostegui.json |
[production] |
| 11:24 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: db952ba: Add zh-hans and zh-hant translation of Module and Module_talk aliases for all Zh Projects (T165593) (duration: 00m 56s) |
[production] |
| 11:16 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 1301fd4: Add import sources for gomwiktionary (T255098) (duration: 00m 57s) |
[production] |
| 11:08 |
<marostegui@cumin2001> |
dbctl commit (dc=all): 'Slowly repool db1094', diff saved to https://phabricator.wikimedia.org/P11622 and previous config saved to /var/cache/conftool/dbconfig/20200622-110806-marostegui.json |
[production] |
| 11:07 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: defa81e: Disable NS_USER(_TALK) search engine indexing on trwiki (T255538) (duration: 00m 58s) |
[production] |
| 10:35 |
<jdrewniak@deploy1001> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:606985| Bumping portals to master (606985)]] (duration: 00m 56s) |
[production] |
| 10:34 |
<jdrewniak@deploy1001> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:606985| Bumping portals to master (606985)]] (duration: 01m 12s) |
[production] |
| 09:58 |
<marostegui@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:56 |
<marostegui@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:33 |
<marostegui@cumin2001> |
dbctl commit (dc=all): 'Depool db1094 for reimage', diff saved to https://phabricator.wikimedia.org/P11621 and previous config saved to /var/cache/conftool/dbconfig/20200622-093323-marostegui.json |
[production] |
| 09:31 |
<godog> |
roll-restart logstash in codfw/eqiad to apply configuration change |
[production] |
| 08:59 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:56 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:33 |
<moritzm> |
reimaging cumin1001 to buster T245114 |
[production] |
| 08:13 |
<godog> |
extend prometheus codfw ops filesystem to 1TB |
[production] |
| 08:02 |
<vgutierrez> |
upgrade to trafficserver 8.0.8~rc0-1wm1 on cp4026 and cp4032 |
[production] |
| 08:02 |
<vgutierrez> |
upload trafficserver 8.0.8~rc0-1wm1 to apt.wm.o (buster) |
[production] |
| 07:33 |
<marostegui@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:30 |
<marostegui@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:16 |
<marostegui> |
Reimage db1117 (irc haproxy alerts will be triggered) |
[production] |
| 06:26 |
<marostegui@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:24 |
<marostegui@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:06 |
<marostegui> |
Stop MySQL on dbstore1005 for reimage to Buster - T254870 |
[production] |
| 05:58 |
<marostegui> |
Compress InnoDb on db1118 T254462 |
[production] |
| 05:51 |
<marostegui@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 05:49 |
<marostegui@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 05:43 |
<marostegui> |
Stop haproxy on dbproxy1008 - T255406 |
[production] |