|
2020-09-23
§
|
| 10:20 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db2084 after index changes T262856', diff saved to https://phabricator.wikimedia.org/P12751 and previous config saved to /var/cache/conftool/dbconfig/20200923-100156-marostegui.json |
[production] |
| 10:01 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/Wikibase.php: Config: [[gerrit:629133|Configure entityDataCachePaths for Wikibase]] (duration: 01m 05s) |
[production] |
| 09:59 |
<elukey> |
update puppet compilter's facts |
[production] |
| 09:57 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: Config: [[gerrit:620050|Remove $wgExtraLanguageNames from Wikidata and Commons (T260118)]], part 2/2 (production no-op) (duration: 01m 04s) |
[production] |
| 09:55 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:620050|Remove $wgExtraLanguageNames from Wikidata and Commons (T260118)]], part 1/2 (duration: 01m 16s) |
[production] |
| 09:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2084 after index changes T262856', diff saved to https://phabricator.wikimedia.org/P12750 and previous config saved to /var/cache/conftool/dbconfig/20200923-094511-marostegui.json |
[production] |
| 08:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2084 after index changes T262856', diff saved to https://phabricator.wikimedia.org/P12748 and previous config saved to /var/cache/conftool/dbconfig/20200923-083200-marostegui.json |
[production] |
| 08:29 |
<moritzm> |
installing dbus security updates on buster |
[production] |
| 08:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2084 after index changes T262856', diff saved to https://phabricator.wikimedia.org/P12747 and previous config saved to /var/cache/conftool/dbconfig/20200923-080651-marostegui.json |
[production] |
| 07:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2084 after index changes T262856', diff saved to https://phabricator.wikimedia.org/P12746 and previous config saved to /var/cache/conftool/dbconfig/20200923-071129-marostegui.json |
[production] |
| 07:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2084 to re-add change_revision_id index T262856', diff saved to https://phabricator.wikimedia.org/P12745 and previous config saved to /var/cache/conftool/dbconfig/20200923-070926-marostegui.json |
[production] |
| 06:34 |
<marostegui> |
Stop MySQL on es2012 and es2018 T263613 T263615 |
[production] |
| 06:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es2018 T263615', diff saved to https://phabricator.wikimedia.org/P12744 and previous config saved to /var/cache/conftool/dbconfig/20200923-063140-marostegui.json |
[production] |
| 06:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es2012 for decommmissioning', diff saved to https://phabricator.wikimedia.org/P12743 and previous config saved to /var/cache/conftool/dbconfig/20200923-060812-marostegui.json |
[production] |
| 05:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db2084 after index removal T262856', diff saved to https://phabricator.wikimedia.org/P12742 and previous config saved to /var/cache/conftool/dbconfig/20200923-055850-marostegui.json |
[production] |
| 05:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2084 T262856', diff saved to https://phabricator.wikimedia.org/P12741 and previous config saved to /var/cache/conftool/dbconfig/20200923-055531-marostegui.json |
[production] |
| 05:37 |
<marostegui> |
Purge global_status_log table on tendril - T252331 |
[production] |
| 05:33 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 05:16 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 05:03 |
<marostegui> |
Remove triggers from db2094:3313 for MCR schema change T238966 |
[production] |
| 05:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2074 for MCR schema change', diff saved to https://phabricator.wikimedia.org/P12739 and previous config saved to /var/cache/conftool/dbconfig/20200923-050234-marostegui.json |
[production] |
| 04:25 |
<eileen> |
civicrm revision changed from 8f32b6301f to a789afd79b, config revision is 9933605187 |
[production] |
|
2020-09-22
§
|
| 23:27 |
<legoktm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: clientError: enable on ja,es,de,ru,it,zh,pt wikipedias (T255585) (duration: 01m 04s) |
[production] |
| 23:23 |
<legoktm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Enable watchlist expiry feature (T261249) (duration: 01m 06s) |
[production] |
| 21:50 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 21:48 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:46 |
<ebernhardson> |
T259539 enabled adaptive replica selection on elasticsearch at search.svc.eqiad.wmnet:9[246]43 |
[production] |
| 20:44 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:43 |
<dancy@deploy1001> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.36.0-wmf.10 |
[production] |
| 20:42 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:31 |
<dancy@deploy1001> |
Finished scap: testwikis wikis to 1.36.0-wmf.10 (duration: 42m 21s) |
[production] |
| 20:30 |
<mutante> |
gerrit2001 (gerrit-replica) restarting gerrit service |
[production] |
| 19:49 |
<dancy@deploy1001> |
Started scap: testwikis wikis to 1.36.0-wmf.10 |
[production] |
| 19:44 |
<dancy@deploy1001> |
Pruned MediaWiki: 1.36.0-wmf.5 (duration: 17m 59s) |
[production] |
| 19:31 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 19:29 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:26 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:24 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:52 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 16:50 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:38 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) |
[production] |
| 16:20 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 16:18 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:00 |
<robh> |
running dell epsa test on down host mw1360 per T262151 |
[production] |
| 14:34 |
<moritzm> |
installing nginx security updates on buster |
[production] |
| 14:33 |
<shdubsh> |
restart apache on prometheus nodes to pick up new ext endpoint |
[production] |
| 14:24 |
<ema> |
upload libvmod-re2 1.5.3-1 to buster-wikimedia component/varnish6 T261632 |
[production] |
| 14:24 |
<papaul> |
rebooting ms-be2019 |
[production] |
| 14:15 |
<XioNoX> |
upgrade FNM on netflow2001 - T257035 |
[production] |