|
2023-03-28
§
|
| 08:05 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on 21 hosts with reason: Switch maintenance |
[production] |
| 08:04 |
<oblivian@deploy2002> |
oblivian and filippo: Backport for [[gerrit:903209|Failover statsd to graphite2004 (T330165)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 08:03 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on es[1020-1022].eqiad.wmnet with reason: Switch maintenance |
[production] |
| 08:03 |
<ayounsi@deploy1002> |
helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 08:03 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on es[1020-1022].eqiad.wmnet with reason: Switch maintenance |
[production] |
| 08:02 |
<oblivian@deploy2002> |
Started scap: Backport for [[gerrit:903209|Failover statsd to graphite2004 (T330165)]] |
[production] |
| 08:02 |
<ayounsi@deploy1002> |
helmfile [staging-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 08:00 |
<ayounsi@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 08:00 |
<godog> |
move graphite reads to codfw - T330165 |
[production] |
| 07:56 |
<jayme@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 07:56 |
<jayme@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 07:56 |
<ayounsi@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 07:54 |
<root@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 07:54 |
<root@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 07:51 |
<ayounsi@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 07:51 |
<ayounsi@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 07:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1179 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P45965 and previous config saved to /var/cache/conftool/dbconfig/20230328-073122-root.json |
[production] |
| 07:28 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'clear' for AS: 17806 |
[production] |
| 07:27 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'clear' for AS: 17806 |
[production] |
| 07:20 |
<kartik@deploy2002> |
Finished scap: Backport for [[gerrit:903003|Enable Section Translation on some wikis while Content Translation remains in beta (T308834)]] (duration: 12m 05s) |
[production] |
| 07:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1179 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P45964 and previous config saved to /var/cache/conftool/dbconfig/20230328-071617-root.json |
[production] |
| 07:10 |
<kartik@deploy2002> |
kartik: Backport for [[gerrit:903003|Enable Section Translation on some wikis while Content Translation remains in beta (T308834)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 07:08 |
<kartik@deploy2002> |
Started scap: Backport for [[gerrit:903003|Enable Section Translation on some wikis while Content Translation remains in beta (T308834)]] |
[production] |
| 07:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1179 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P45963 and previous config saved to /var/cache/conftool/dbconfig/20230328-070112-root.json |
[production] |
| 06:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1179 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P45962 and previous config saved to /var/cache/conftool/dbconfig/20230328-064607-root.json |
[production] |
| 06:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1179 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P45961 and previous config saved to /var/cache/conftool/dbconfig/20230328-063103-root.json |
[production] |
| 06:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1179 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P45960 and previous config saved to /var/cache/conftool/dbconfig/20230328-061558-root.json |
[production] |
| 06:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1104 T329481', diff saved to https://phabricator.wikimedia.org/P45959 and previous config saved to /var/cache/conftool/dbconfig/20230328-061441-root.json |
[production] |
| 06:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1179 (re)pooling @ 4%: Repooling', diff saved to https://phabricator.wikimedia.org/P45958 and previous config saved to /var/cache/conftool/dbconfig/20230328-060053-root.json |
[production] |
| 05:55 |
<oblivian@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 05:55 |
<oblivian@deploy2002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 05:53 |
<oblivian@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 05:53 |
<oblivian@deploy2002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 05:47 |
<AndyRussG> |
update payments-wiki f5e262d1 -> a6c6c2b1 |
[production] |
| 05:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1179 (re)pooling @ 3%: Repooling', diff saved to https://phabricator.wikimedia.org/P45957 and previous config saved to /var/cache/conftool/dbconfig/20230328-054548-root.json |
[production] |
| 05:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1179 (re)pooling @ 2%: Repooling', diff saved to https://phabricator.wikimedia.org/P45956 and previous config saved to /var/cache/conftool/dbconfig/20230328-053043-root.json |
[production] |
| 05:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1179 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P45955 and previous config saved to /var/cache/conftool/dbconfig/20230328-051539-root.json |
[production] |
| 01:59 |
<krinkle@deploy2002> |
Synchronized wmf-config/mc.php: I44edcd46da45b827d (duration: 06m 33s) |
[production] |
|
2023-03-27
§
|
| 23:47 |
<mutante> |
people1003 - taking down apache to provoke monitoring alert (inactive instances) and confirm IRC alerting change works |
[production] |
| 23:31 |
<zabe> |
deployed patch for T330968 |
[production] |
| 23:08 |
<zabe@deploy2002> |
Finished scap: Backport for [[gerrit:903205|Rename "Support and Safety" to "Trust and Safety" (T330514)]] (duration: 21m 27s) |
[production] |
| 23:00 |
<zabe@deploy2002> |
zabe: Backport for [[gerrit:903205|Rename "Support and Safety" to "Trust and Safety" (T330514)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 22:48 |
<mutante> |
stat1005 - kill 18179; run puppet ; stat1007 - kill 3346; run puppet ; stat1006 - kill 23887 run puppet |
[production] |
| 22:47 |
<zabe@deploy2002> |
Started scap: Backport for [[gerrit:903205|Rename "Support and Safety" to "Trust and Safety" (T330514)]] |
[production] |
| 22:43 |
<mutante> |
stat1004 - kill 29291; run puppet |
[production] |
| 22:43 |
<mutante> |
apt2001 - kill 3105; run puppet |
[production] |
| 22:16 |
<zabe> |
zabe@mwmaint2002:~$ mwscript extensions/Translate/scripts/moveTranslatableBundle.php --wiki metawiki "Meta:WMF Support and Safety" "Meta:WMF Trust and Safety" "Zabe" --reason "per [[:phab:T330514|T330514]]" # T330514 |
[production] |
| 21:58 |
<maryum> |
Deploy security fix for T326952 |
[production] |
| 21:58 |
<urandom> |
power cycling restbase1033 — T333243 |
[production] |
| 21:45 |
<ryankemper> |
T330165 Depooled relevant search platform hosts: `sudo -E cumin 'elastic[1055-1056,1074-1079,1085-1086]*,cloudelastic100[2,6]*,wcqs1002*,wdqs[1007,1012]*' 'sudo depool'` |
[production] |