|
2022-01-24
ยง
|
| 14:44 |
<elukey@cumin1001> |
START - Cookbook sre.ores.roll-restart-workers for ORES codfw cluster: Roll restart of ORES's daemons. |
[production] |
| 14:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1172 (T285149)', diff saved to https://phabricator.wikimedia.org/P19059 and previous config saved to /var/cache/conftool/dbconfig/20220124-144234-marostegui.json |
[production] |
| 14:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 75%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19058 and previous config saved to /var/cache/conftool/dbconfig/20220124-144208-root.json |
[production] |
| 14:34 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2034.codfw.wmnet with OS bullseye |
[production] |
| 14:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 60%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19057 and previous config saved to /var/cache/conftool/dbconfig/20220124-142705-root.json |
[production] |
| 14:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 50%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19056 and previous config saved to /var/cache/conftool/dbconfig/20220124-141201-root.json |
[production] |
| 14:01 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.reimage for host restbase1030.eqiad.wmnet with OS buster |
[production] |
| 14:00 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=restbase1029.eqiad.wmnet |
[production] |
| 14:00 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.reimage for host es2034.codfw.wmnet with OS bullseye |
[production] |
| 14:00 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host restbase1029.eqiad.wmnet with OS buster |
[production] |
| 13:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 40%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19055 and previous config saved to /var/cache/conftool/dbconfig/20220124-135658-root.json |
[production] |
| 13:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1172 (T285149)', diff saved to https://phabricator.wikimedia.org/P19054 and previous config saved to /var/cache/conftool/dbconfig/20220124-135216-marostegui.json |
[production] |
| 13:52 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1172.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:52 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1172.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1126 (T285149)', diff saved to https://phabricator.wikimedia.org/P19053 and previous config saved to /var/cache/conftool/dbconfig/20220124-135208-marostegui.json |
[production] |
| 13:50 |
<moritzm> |
installing util-linux security updates on bullseye |
[production] |
| 13:42 |
<ladsgroup@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host es2034.codfw.wmnet with OS bullseye |
[production] |
| 13:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 25%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19052 and previous config saved to /var/cache/conftool/dbconfig/20220124-134154-root.json |
[production] |
| 13:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1126', diff saved to https://phabricator.wikimedia.org/P19051 and previous config saved to /var/cache/conftool/dbconfig/20220124-133704-marostegui.json |
[production] |
| 13:28 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=restbase1028.eqiad.wmnet |
[production] |
| 13:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 20%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19050 and previous config saved to /var/cache/conftool/dbconfig/20220124-132651-root.json |
[production] |
| 13:26 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.reimage for host restbase1029.eqiad.wmnet with OS buster |
[production] |
| 13:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1126', diff saved to https://phabricator.wikimedia.org/P19049 and previous config saved to /var/cache/conftool/dbconfig/20220124-132159-marostegui.json |
[production] |
| 13:19 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host restbase1028.eqiad.wmnet with OS buster |
[production] |
| 13:13 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.reimage for host es2034.codfw.wmnet with OS bullseye |
[production] |
| 13:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 10%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19048 and previous config saved to /var/cache/conftool/dbconfig/20220124-131147-root.json |
[production] |
| 13:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1126 (T285149)', diff saved to https://phabricator.wikimedia.org/P19047 and previous config saved to /var/cache/conftool/dbconfig/20220124-130654-marostegui.json |
[production] |
| 13:06 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2034.codfw.wmnet with reason: reimage for upgrade - T299911 |
[production] |
| 13:06 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on es2034.codfw.wmnet with reason: reimage for upgrade - T299911 |
[production] |
| 12:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 5%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19046 and previous config saved to /var/cache/conftool/dbconfig/20220124-125643-root.json |
[production] |
| 12:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 1%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19045 and previous config saved to /var/cache/conftool/dbconfig/20220124-124140-root.json |
[production] |
| 12:41 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 12:40 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1033.eqiad.wmnet with OS bullseye |
[production] |
| 12:40 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.reimage for host restbase1028.eqiad.wmnet with OS buster |
[production] |
| 12:39 |
<hnowlan@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host restbase1027.eqiad.wmnet with OS buster |
[production] |
| 12:39 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=restbase1027.eqiad.wmnet |
[production] |
| 12:39 |
<ayounsi@cumin1001> |
END (FAIL) - Cookbook sre.network.prepare-upgrade (exit_code=99) |
[production] |
| 12:39 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.prepare-upgrade |
[production] |
| 12:39 |
<ayounsi@cumin1001> |
END (FAIL) - Cookbook sre.network.prepare-upgrade (exit_code=99) |
[production] |
| 12:38 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.prepare-upgrade |
[production] |
| 12:36 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 12:36 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 12:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 100%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19044 and previous config saved to /var/cache/conftool/dbconfig/20220124-123609-root.json |
[production] |
| 12:32 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 12:27 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 12:25 |
<urbanecm> |
UTC morning B&C done |
[production] |
| 12:25 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.reimage for host restbase1027.eqiad.wmnet with OS buster |
[production] |
| 12:24 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=restbase1026.eqiad.wmnet |
[production] |
| 12:24 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 296fe1644a2a71914e880f3562f8e32fd66c1637: Add mwcli.command_execute to wgEventStreams (T293583) (duration: 00m 48s) |
[production] |
| 12:22 |
<hnowlan@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host restbase1026.eqiad.wmnet with OS buster |
[production] |