|
2020-05-21
§
|
| 15:07 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 15:07 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:59 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw217[0-2].codfw.wmnet |
[production] |
| 14:59 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw216[0-9].codfw.wmnet |
[production] |
| 14:58 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw215[8-9].codfw.wmnet |
[production] |
| 14:50 |
<bblack@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:47 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:44 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'mathoid' for release 'canary' . |
[production] |
| 14:33 |
<akosiaris> |
upload helmfile 0.109.0 to apt.wikimedia.org/buster-wikimedia and stretch-wikimedia, component main |
[production] |
| 13:51 |
<vgutierrez> |
depool cp4032 for some ats tests |
[production] |
| 13:22 |
<mutante> |
cloudnet1004 - reboot to test PXE boot |
[production] |
| 12:44 |
<andrewbogott> |
reimaging cloudnet1004.eqiad.wmnet for T253124 |
[production] |
| 12:29 |
<elukey> |
roll restart druid-public cluster (druid100[4-6], backend for the AQS API) to apply new settings + openjdk upgrade - T252771 |
[production] |
| 12:13 |
<mutante> |
depooled mw2158 through mw2172 to make room again in C3 as planned (T247018) |
[production] |
| 12:12 |
<marostegui> |
Repool labsdb1011 into the analytics role 🤞- T249188 |
[production] |
| 12:12 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw217[0-2].codfw.wmnet |
[production] |
| 12:10 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw216[0-9].codfw.wmnet |
[production] |
| 12:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1143 and db1091', diff saved to https://phabricator.wikimedia.org/P11270 and previous config saved to /var/cache/conftool/dbconfig/20200521-120555-marostegui.json |
[production] |
| 12:05 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw215[8-9].codfw.wmnet |
[production] |
| 11:18 |
<hnowlan> |
Removed changeprop from scb hosts |
[production] |
| 11:04 |
<vgutierrez> |
rolling restart of ncredir servers for kernel update |
[production] |
| 10:17 |
<vgutierrez> |
restart of acme-chief servers for kernel update |
[production] |
| 10:13 |
<jbond42> |
deploy CI for pupet privcate repo |
[production] |
| 10:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1143 and db1091', diff saved to https://phabricator.wikimedia.org/P11268 and previous config saved to /var/cache/conftool/dbconfig/20200521-101100-marostegui.json |
[production] |
| 10:07 |
<mutante> |
replaced backend of people.wikimedia.org - people1001 will be inaccessible, replaced with people1002 on buster. all home dirs have been synced over, there should be no difference except you have to use people1002 now for uploads (T247649) |
[production] |
| 10:06 |
<godog> |
test adding --sni to check_http -S on icinga2001 - T253292 |
[production] |
| 09:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1143 and db1091', diff saved to https://phabricator.wikimedia.org/P11267 and previous config saved to /var/cache/conftool/dbconfig/20200521-095100-marostegui.json |
[production] |
| 09:28 |
<mutante> |
deneb - sudo systemctl reset-failed to clear Icinga alerts about systemd degraded state |
[production] |
| 09:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1143 and db1091', diff saved to https://phabricator.wikimedia.org/P11266 and previous config saved to /var/cache/conftool/dbconfig/20200521-091245-marostegui.json |
[production] |
| 09:01 |
<mutante> |
LDAP - added lmata to wmf group (T253277) |
[production] |
| 08:55 |
<XioNoX> |
Advertise Anycast 198.35.27.0/24 from esams - T253196 |
[production] |
| 08:52 |
<XioNoX> |
Advertise Anycast 198.35.27.0/24 from eqsin - T253196 |
[production] |
| 08:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool db1143 with minimal weight for the first time T252512', diff saved to https://phabricator.wikimedia.org/P11265 and previous config saved to /var/cache/conftool/dbconfig/20200521-084933-marostegui.json |
[production] |
| 08:47 |
<XioNoX> |
Advertise Anycast 198.35.27.0/24 from eqiad/eqord - T253196 |
[production] |
| 08:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db1143 to the list of s4 hosts, depooled - T252512', diff saved to https://phabricator.wikimedia.org/P11264 and previous config saved to /var/cache/conftool/dbconfig/20200521-084226-marostegui.json |
[production] |
| 08:34 |
<XioNoX> |
Advertise Anycast 198.35.27.0/24 from dfw - T253196 |
[production] |
| 08:27 |
<XioNoX> |
Advertise Anycast 198.35.27.0/24 from ulsfo - T253196 |
[production] |
| 08:20 |
<XioNoX> |
Delete ARIN route object for 198.35.26.0/23 - T253196 |
[production] |
| 08:13 |
<XioNoX> |
Delete ROA for 198.35.26.0/23 - T253196 |
[production] |
| 08:10 |
<XioNoX> |
repool ulsfo - T253196 |
[production] |
| 08:03 |
<XioNoX> |
Shrink ulsfo's 198.35.26.0/23 to 198.35.26.0/24 - T253196 |
[production] |
| 07:29 |
<XioNoX> |
depool ulsfo - T253196 |
[production] |
| 07:22 |
<marostegui> |
Purge events from tendril.global_status_log older than 24h - T252331 |
[production] |
| 07:03 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Repool es1019 fully', diff saved to https://phabricator.wikimedia.org/P11263 and previous config saved to /var/cache/conftool/dbconfig/20200521-070335-jynus.json |
[production] |
| 06:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1091 - T252512', diff saved to https://phabricator.wikimedia.org/P11261 and previous config saved to /var/cache/conftool/dbconfig/20200521-065858-marostegui.json |
[production] |
| 06:28 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Repool es1019 with 50% weight', diff saved to https://phabricator.wikimedia.org/P11260 and previous config saved to /var/cache/conftool/dbconfig/20200521-062823-jynus.json |
[production] |
| 06:04 |
<vgutierrez> |
pool cp5012 - T251219 |
[production] |
| 05:42 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Repool es1019 with low weight', diff saved to https://phabricator.wikimedia.org/P11259 and previous config saved to /var/cache/conftool/dbconfig/20200521-054231-jynus.json |
[production] |
| 05:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set enwiki as read-only=off after maintenance T251982', diff saved to https://phabricator.wikimedia.org/P11258 and previous config saved to /var/cache/conftool/dbconfig/20200521-050328-marostegui.json |
[production] |
| 05:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set enwiki as read-only for maintenance T251982', diff saved to https://phabricator.wikimedia.org/P11257 and previous config saved to /var/cache/conftool/dbconfig/20200521-050029-marostegui.json |
[production] |