|
2020-02-18
§
|
| 11:48 |
<kartik@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 11:47 |
<kartik@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 11:43 |
<kartik@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 11:41 |
<kartik@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 11:35 |
<kartik@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 11:27 |
<jynus> |
reenabling prometheus exporter metadata user for prometheus1003 |
[production] |
| 11:10 |
<jynus> |
temp. disabling prometheus exporter metadata user for prometheus1003 |
[production] |
| 10:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase API weight for db1107 15 -> 25 for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10445 and previous config saved to /var/cache/conftool/dbconfig/20200218-104958-marostegui.json |
[production] |
| 09:27 |
<gehel> |
re-enable puppet on mw* - T222321 |
[production] |
| 09:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1107 after temporary change optimizer options - T245489', diff saved to https://phabricator.wikimedia.org/P10444 and previous config saved to /var/cache/conftool/dbconfig/20200218-091343-marostegui.json |
[production] |
| 09:09 |
<gehel> |
disabling puppet on mw* to deploy apache config change - T222321 |
[production] |
| 09:07 |
<volans> |
rm /var/log/exim4/paniclog on cumin1001 to clear OOM from last week error |
[production] |
| 08:59 |
<marostegui> |
Remove wikiadmin2 grants from es1 T243512 |
[production] |
| 08:59 |
<marostegui> |
Remove wikiadmin2 grants from es1 |
[production] |
| 08:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1107 after temporary change optimizer options', diff saved to https://phabricator.wikimedia.org/P10443 and previous config saved to /var/cache/conftool/dbconfig/20200218-085713-marostegui.json |
[production] |
| 08:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1107 after temporary change optimizer options - T245489', diff saved to https://phabricator.wikimedia.org/P10442 and previous config saved to /var/cache/conftool/dbconfig/20200218-082306-marostegui.json |
[production] |
| 08:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1107 after temporary change optimizer options - T245489', diff saved to https://phabricator.wikimedia.org/P10441 and previous config saved to /var/cache/conftool/dbconfig/20200218-080952-marostegui.json |
[production] |
| 08:08 |
<marostegui> |
Restart MySQL to pick up optimizer_switch changes - T245489 |
[production] |
| 08:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1107 to temporary change optimizer options - T245489', diff saved to https://phabricator.wikimedia.org/P10440 and previous config saved to /var/cache/conftool/dbconfig/20200218-080623-marostegui.json |
[production] |
| 07:34 |
<elukey> |
powercycle analytics1065 (crashed hours ago, no mgmt console available, no ssh) |
[production] |
| 06:39 |
<marostegui> |
Remove wikiadmin2 from pc1007, pc1008, pc1009 and pc1010 T243512 |
[production] |
| 06:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase weight for db1107 100 -> 200 for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10439 and previous config saved to /var/cache/conftool/dbconfig/20200218-063819-marostegui.json |
[production] |
| 06:27 |
<marostegui> |
Stop haproxy on dbproxy1007 - T245385 |
[production] |
| 06:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1107 with weight 100 and weight 10 in API for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10438 and previous config saved to /var/cache/conftool/dbconfig/20200218-062459-marostegui.json |
[production] |
| 06:09 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 06:08 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
|
2020-02-17
§
|
| 19:56 |
<cdanis> |
finish enabling TCP-MSS clamping in eqiad |
[production] |
| 19:49 |
<cdanis> |
s/no-op// |
[production] |
| 19:49 |
<cdanis> |
no-op enable TCP-MSS clamping on eqord and eqiad |
[production] |
| 19:33 |
<cdanis> |
no-op enable flowspec change on cr2-eqord and cr2-eqiad |
[production] |
| 18:25 |
<elukey> |
restart kafka on kafka-jumbo1001 to pick up new openjdk updates |
[production] |
| 17:25 |
<bblack> |
GRE MTU mitigations applied to esams cp hosts only - T232602 |
[production] |
| 15:55 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 15:50 |
<ayounsi@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 15:48 |
<ayounsi@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 15:48 |
<ayounsi@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 15:44 |
<cdanis> |
✔️ cdanis@icinga1001.wikimedia.org ~ 🕥☕ sudo systemctl restart ircecho |
[production] |
| 14:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1107 after 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10422 and previous config saved to /var/cache/conftool/dbconfig/20200217-143146-marostegui.json |
[production] |
| 14:17 |
<ema> |
reprepro includedeb buster-wikimedia ~ema/cadvisor_0.35.0+ds1-4_amd64.deb T183146 |
[production] |
| 12:34 |
<XioNoX> |
add test flowspec rules to cr3-knams |
[production] |
| 12:34 |
<moritzm> |
installing postgresql-9.4 security updates |
[production] |
| 12:27 |
<vgutierrez> |
reboot acmechief instances (kernel upgrade) |
[production] |
| 10:31 |
<jynus> |
dropping all databases from db1140:3313 |
[production] |
| 10:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): ' db1107 increase API weight from 10 to 15 for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10420 and previous config saved to /var/cache/conftool/dbconfig/20200217-102218-marostegui.json |
[production] |
| 10:20 |
<vgutierrez> |
rolling restart of ats-tls and varnish-fe on ulsfo to enable KA between them - T244464 |
[production] |
| 10:00 |
<moritzm> |
installing Linux 4.9.210 kernels on stretch systems |
[production] |
| 09:10 |
<godog> |
correction, +100G |
[production] |
| 09:09 |
<godog> |
+10G to prometheus/ops fs on prometheus eqiad - T245361 |
[production] |
| 09:06 |
<godog> |
+50G to prometheus/ops fs on prometheus eqiad - T245361 |
[production] |
| 07:22 |
<marostegui> |
Stop haproxy on dbproxy1002 - T245384 |
[production] |