|
2020-02-18
§
|
| 12:58 |
<kartik@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 12:55 |
<Amir1> |
EU SWAT done |
[production] |
| 12:53 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:572731|Add DiscussionTools to four wikis in hidden mode (T244870)]], take II (duration: 01m 03s) |
[production] |
| 12:52 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:572731|Add DiscussionTools to four wikis in hidden mode (T244870)]] (duration: 01m 04s) |
[production] |
| 12:45 |
<XioNoX> |
remove graceful-switchover and nonstop-routing from cr2-esams - T243080 |
[production] |
| 12:36 |
<XioNoX> |
push new Junos to cr2-esams:re1 (backup RE, noop) - T243080 |
[production] |
| 12:22 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/Wikibase.php: SWAT: [[gerrit:569031|Wikibase: added config variables to configure entity sources (T242087)]], Part II (duration: 01m 03s) |
[production] |
| 12:20 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:569031|Wikibase: added config variables to configure entity sources (T242087)]], Part I, take II (the cache issue) (duration: 01m 04s) |
[production] |
| 12:18 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:569031|Wikibase: added config variables to configure entity sources (T242087)]], Part I (duration: 01m 06s) |
[production] |
| 12:14 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:572628|Start reading for the new term store for clients up to Q1000 (T225057)]] (duration: 01m 05s) |
[production] |
| 12:06 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 4b193dd: Increase Commons linkpurge rate limit for patrollers (T245214) (duration: 01m 31s) |
[production] |
| 11:51 |
<kartik@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 11:48 |
<kartik@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 11:47 |
<kartik@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 11:43 |
<kartik@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 11:41 |
<kartik@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 11:35 |
<kartik@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 11:27 |
<jynus> |
reenabling prometheus exporter metadata user for prometheus1003 |
[production] |
| 11:10 |
<jynus> |
temp. disabling prometheus exporter metadata user for prometheus1003 |
[production] |
| 10:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase API weight for db1107 15 -> 25 for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10445 and previous config saved to /var/cache/conftool/dbconfig/20200218-104958-marostegui.json |
[production] |
| 09:27 |
<gehel> |
re-enable puppet on mw* - T222321 |
[production] |
| 09:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1107 after temporary change optimizer options - T245489', diff saved to https://phabricator.wikimedia.org/P10444 and previous config saved to /var/cache/conftool/dbconfig/20200218-091343-marostegui.json |
[production] |
| 09:09 |
<gehel> |
disabling puppet on mw* to deploy apache config change - T222321 |
[production] |
| 09:07 |
<volans> |
rm /var/log/exim4/paniclog on cumin1001 to clear OOM from last week error |
[production] |
| 08:59 |
<marostegui> |
Remove wikiadmin2 grants from es1 T243512 |
[production] |
| 08:59 |
<marostegui> |
Remove wikiadmin2 grants from es1 |
[production] |
| 08:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1107 after temporary change optimizer options', diff saved to https://phabricator.wikimedia.org/P10443 and previous config saved to /var/cache/conftool/dbconfig/20200218-085713-marostegui.json |
[production] |
| 08:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1107 after temporary change optimizer options - T245489', diff saved to https://phabricator.wikimedia.org/P10442 and previous config saved to /var/cache/conftool/dbconfig/20200218-082306-marostegui.json |
[production] |
| 08:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1107 after temporary change optimizer options - T245489', diff saved to https://phabricator.wikimedia.org/P10441 and previous config saved to /var/cache/conftool/dbconfig/20200218-080952-marostegui.json |
[production] |
| 08:08 |
<marostegui> |
Restart MySQL to pick up optimizer_switch changes - T245489 |
[production] |
| 08:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1107 to temporary change optimizer options - T245489', diff saved to https://phabricator.wikimedia.org/P10440 and previous config saved to /var/cache/conftool/dbconfig/20200218-080623-marostegui.json |
[production] |
| 07:34 |
<elukey> |
powercycle analytics1065 (crashed hours ago, no mgmt console available, no ssh) |
[production] |
| 06:39 |
<marostegui> |
Remove wikiadmin2 from pc1007, pc1008, pc1009 and pc1010 T243512 |
[production] |
| 06:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase weight for db1107 100 -> 200 for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10439 and previous config saved to /var/cache/conftool/dbconfig/20200218-063819-marostegui.json |
[production] |
| 06:27 |
<marostegui> |
Stop haproxy on dbproxy1007 - T245385 |
[production] |
| 06:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1107 with weight 100 and weight 10 in API for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10438 and previous config saved to /var/cache/conftool/dbconfig/20200218-062459-marostegui.json |
[production] |
| 06:09 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 06:08 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
|
2020-02-17
§
|
| 19:56 |
<cdanis> |
finish enabling TCP-MSS clamping in eqiad |
[production] |
| 19:49 |
<cdanis> |
s/no-op// |
[production] |
| 19:49 |
<cdanis> |
no-op enable TCP-MSS clamping on eqord and eqiad |
[production] |
| 19:33 |
<cdanis> |
no-op enable flowspec change on cr2-eqord and cr2-eqiad |
[production] |
| 18:25 |
<elukey> |
restart kafka on kafka-jumbo1001 to pick up new openjdk updates |
[production] |
| 17:25 |
<bblack> |
GRE MTU mitigations applied to esams cp hosts only - T232602 |
[production] |
| 15:55 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 15:50 |
<ayounsi@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 15:48 |
<ayounsi@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 15:48 |
<ayounsi@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 15:44 |
<cdanis> |
✔️ cdanis@icinga1001.wikimedia.org ~ 🕥☕ sudo systemctl restart ircecho |
[production] |
| 14:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1107 after 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10422 and previous config saved to /var/cache/conftool/dbconfig/20200217-143146-marostegui.json |
[production] |