|
2020-02-18
§
|
| 14:01 |
<XioNoX> |
re-enable cr2-esams BGP group IX4 - T243080 |
[production] |
| 13:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase API weight for db1107 25 -> 50 for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10448 and previous config saved to /var/cache/conftool/dbconfig/20200218-135525-marostegui.json |
[production] |
| 13:44 |
<XioNoX> |
installing OS on cr2-esams:re0 - T243080 |
[production] |
| 13:39 |
<XioNoX> |
cr2-esams - request chassis routing-engine master switch - T243080 |
[production] |
| 13:37 |
<XioNoX> |
deactivate peering/transit on cr2-esams - T243080 |
[production] |
| 13:24 |
<XioNoX> |
reboot cr2-esams:re1 (backup) - T243080 |
[production] |
| 13:23 |
<XioNoX> |
bump cost of eqiad-esams transport - T243080 |
[production] |
| 13:10 |
<XioNoX> |
fail vrrp master to cr3-esams - T243080 |
[production] |
| 12:58 |
<kartik@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 12:55 |
<Amir1> |
EU SWAT done |
[production] |
| 12:53 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:572731|Add DiscussionTools to four wikis in hidden mode (T244870)]], take II (duration: 01m 03s) |
[production] |
| 12:52 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:572731|Add DiscussionTools to four wikis in hidden mode (T244870)]] (duration: 01m 04s) |
[production] |
| 12:45 |
<XioNoX> |
remove graceful-switchover and nonstop-routing from cr2-esams - T243080 |
[production] |
| 12:36 |
<XioNoX> |
push new Junos to cr2-esams:re1 (backup RE, noop) - T243080 |
[production] |
| 12:22 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/Wikibase.php: SWAT: [[gerrit:569031|Wikibase: added config variables to configure entity sources (T242087)]], Part II (duration: 01m 03s) |
[production] |
| 12:20 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:569031|Wikibase: added config variables to configure entity sources (T242087)]], Part I, take II (the cache issue) (duration: 01m 04s) |
[production] |
| 12:18 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:569031|Wikibase: added config variables to configure entity sources (T242087)]], Part I (duration: 01m 06s) |
[production] |
| 12:14 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:572628|Start reading for the new term store for clients up to Q1000 (T225057)]] (duration: 01m 05s) |
[production] |
| 12:06 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 4b193dd: Increase Commons linkpurge rate limit for patrollers (T245214) (duration: 01m 31s) |
[production] |
| 11:51 |
<kartik@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 11:48 |
<kartik@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 11:47 |
<kartik@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 11:43 |
<kartik@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 11:41 |
<kartik@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 11:35 |
<kartik@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 11:27 |
<jynus> |
reenabling prometheus exporter metadata user for prometheus1003 |
[production] |
| 11:10 |
<jynus> |
temp. disabling prometheus exporter metadata user for prometheus1003 |
[production] |
| 10:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase API weight for db1107 15 -> 25 for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10445 and previous config saved to /var/cache/conftool/dbconfig/20200218-104958-marostegui.json |
[production] |
| 09:27 |
<gehel> |
re-enable puppet on mw* - T222321 |
[production] |
| 09:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1107 after temporary change optimizer options - T245489', diff saved to https://phabricator.wikimedia.org/P10444 and previous config saved to /var/cache/conftool/dbconfig/20200218-091343-marostegui.json |
[production] |
| 09:09 |
<gehel> |
disabling puppet on mw* to deploy apache config change - T222321 |
[production] |
| 09:07 |
<volans> |
rm /var/log/exim4/paniclog on cumin1001 to clear OOM from last week error |
[production] |
| 08:59 |
<marostegui> |
Remove wikiadmin2 grants from es1 T243512 |
[production] |
| 08:59 |
<marostegui> |
Remove wikiadmin2 grants from es1 |
[production] |
| 08:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1107 after temporary change optimizer options', diff saved to https://phabricator.wikimedia.org/P10443 and previous config saved to /var/cache/conftool/dbconfig/20200218-085713-marostegui.json |
[production] |
| 08:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1107 after temporary change optimizer options - T245489', diff saved to https://phabricator.wikimedia.org/P10442 and previous config saved to /var/cache/conftool/dbconfig/20200218-082306-marostegui.json |
[production] |
| 08:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1107 after temporary change optimizer options - T245489', diff saved to https://phabricator.wikimedia.org/P10441 and previous config saved to /var/cache/conftool/dbconfig/20200218-080952-marostegui.json |
[production] |
| 08:08 |
<marostegui> |
Restart MySQL to pick up optimizer_switch changes - T245489 |
[production] |
| 08:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1107 to temporary change optimizer options - T245489', diff saved to https://phabricator.wikimedia.org/P10440 and previous config saved to /var/cache/conftool/dbconfig/20200218-080623-marostegui.json |
[production] |
| 07:34 |
<elukey> |
powercycle analytics1065 (crashed hours ago, no mgmt console available, no ssh) |
[production] |
| 06:39 |
<marostegui> |
Remove wikiadmin2 from pc1007, pc1008, pc1009 and pc1010 T243512 |
[production] |
| 06:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase weight for db1107 100 -> 200 for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10439 and previous config saved to /var/cache/conftool/dbconfig/20200218-063819-marostegui.json |
[production] |
| 06:27 |
<marostegui> |
Stop haproxy on dbproxy1007 - T245385 |
[production] |
| 06:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1107 with weight 100 and weight 10 in API for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10438 and previous config saved to /var/cache/conftool/dbconfig/20200218-062459-marostegui.json |
[production] |
| 06:09 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 06:08 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |