|
2022-01-24
ยง
|
| 13:06 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on es2034.codfw.wmnet with reason: reimage for upgrade - T299911 |
[production] |
| 12:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 5%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19046 and previous config saved to /var/cache/conftool/dbconfig/20220124-125643-root.json |
[production] |
| 12:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 1%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19045 and previous config saved to /var/cache/conftool/dbconfig/20220124-124140-root.json |
[production] |
| 12:41 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 12:40 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1033.eqiad.wmnet with OS bullseye |
[production] |
| 12:40 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.reimage for host restbase1028.eqiad.wmnet with OS buster |
[production] |
| 12:39 |
<hnowlan@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host restbase1027.eqiad.wmnet with OS buster |
[production] |
| 12:39 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=restbase1027.eqiad.wmnet |
[production] |
| 12:39 |
<ayounsi@cumin1001> |
END (FAIL) - Cookbook sre.network.prepare-upgrade (exit_code=99) |
[production] |
| 12:39 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.prepare-upgrade |
[production] |
| 12:39 |
<ayounsi@cumin1001> |
END (FAIL) - Cookbook sre.network.prepare-upgrade (exit_code=99) |
[production] |
| 12:38 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.prepare-upgrade |
[production] |
| 12:36 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 12:36 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 12:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 100%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19044 and previous config saved to /var/cache/conftool/dbconfig/20220124-123609-root.json |
[production] |
| 12:32 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 12:27 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 12:25 |
<urbanecm> |
UTC morning B&C done |
[production] |
| 12:25 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.reimage for host restbase1027.eqiad.wmnet with OS buster |
[production] |
| 12:24 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=restbase1026.eqiad.wmnet |
[production] |
| 12:24 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 296fe1644a2a71914e880f3562f8e32fd66c1637: Add mwcli.command_execute to wgEventStreams (T293583) (duration: 00m 48s) |
[production] |
| 12:22 |
<hnowlan@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host restbase1026.eqiad.wmnet with OS buster |
[production] |
| 12:22 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 12:22 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 12:22 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: db340cc: 5424d69: Update wgCopyUploadsDomains allowlist (T299579, T299881) (duration: 00m 48s) |
[production] |
| 12:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 75%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19043 and previous config saved to /var/cache/conftool/dbconfig/20220124-122106-root.json |
[production] |
| 12:21 |
<moritzm> |
installing ICU security updates on stretch |
[production] |
| 12:19 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 2c7b45a3f080757338a877f4024e27dea8cc47c5: fawiki: Exempt draft namespace from robots control by users (T299850) (duration: 05m 39s) |
[production] |
| 12:18 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 12:14 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.reimage for host restbase1026.eqiad.wmnet with OS buster |
[production] |
| 12:13 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host es1033.eqiad.wmnet with OS bullseye |
[production] |
| 12:13 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 12:11 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 12:11 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 12:10 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=restbase102[45].eqiad.wmnet |
[production] |
| 12:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1033 T299889', diff saved to https://phabricator.wikimedia.org/P19042 and previous config saved to /var/cache/conftool/dbconfig/20220124-121029-marostegui.json |
[production] |
| 12:10 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 97d047bc4bf3748fc76f63647d77d26cc545b49f: Remove kea, nod, and sms from wmfGetVariantSettings (T299304; T296286; T298075; T298182) (duration: 00m 49s) |
[production] |
| 12:10 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 12:08 |
<hnowlan@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host restbase1025.eqiad.wmnet with OS buster |
[production] |
| 12:08 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2026.codfw.wmnet with OS bullseye |
[production] |
| 12:06 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on ganeti1014.eqiad.wmnet with reason: Remove from Ganeti cluster for reimage |
[production] |
| 12:06 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on ganeti1014.eqiad.wmnet with reason: Remove from Ganeti cluster for reimage |
[production] |
| 12:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1126 (T285149)', diff saved to https://phabricator.wikimedia.org/P19041 and previous config saved to /var/cache/conftool/dbconfig/20220124-120635-marostegui.json |
[production] |
| 12:06 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1126.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:06 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1126.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1167 (T285149)', diff saved to https://phabricator.wikimedia.org/P19040 and previous config saved to /var/cache/conftool/dbconfig/20220124-120627-marostegui.json |
[production] |
| 12:06 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 12:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 60%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19039 and previous config saved to /var/cache/conftool/dbconfig/20220124-120602-root.json |
[production] |
| 12:05 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 1a463610ba2a92f7437c6921a9591616de0d242e: fawiki: Remove move-rootuserpages flag from users (T299847) (duration: 00m 49s) |
[production] |
| 12:00 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.reimage for host restbase1025.eqiad.wmnet with OS buster |
[production] |