|
2022-03-24
§
|
| 23:57 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host restbase2027.codfw.wmnet with OS buster |
[production] |
| 23:04 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host restbase2027.mgmt.codfw.wmnet with reboot policy FORCED |
[production] |
| 22:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T302658)', diff saved to https://phabricator.wikimedia.org/P23050 and previous config saved to /var/cache/conftool/dbconfig/20220324-223031-marostegui.json |
[production] |
| 22:19 |
<pt1979@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1047.eqiad.wmnet with OS bullseye |
[production] |
| 22:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P23049 and previous config saved to /var/cache/conftool/dbconfig/20220324-221526-marostegui.json |
[production] |
| 22:14 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.provision for host restbase2027.mgmt.codfw.wmnet with reboot policy FORCED |
[production] |
| 22:10 |
<pt1979@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1047.eqiad.wmnet with reason: host reimage |
[production] |
| 22:07 |
<ebernhardson> |
restart wcqs-blazegraph on wcqs2001 to resolve intermittant BlazegraphFreeAllocatorsDecreasingRapidly |
[production] |
| 22:06 |
<pt1979@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1047.eqiad.wmnet with reason: host reimage |
[production] |
| 22:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P23048 and previous config saved to /var/cache/conftool/dbconfig/20220324-220021-marostegui.json |
[production] |
| 21:54 |
<pt1979@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1047.eqiad.wmnet with OS bullseye |
[production] |
| 21:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T302658)', diff saved to https://phabricator.wikimedia.org/P23047 and previous config saved to /var/cache/conftool/dbconfig/20220324-214515-marostegui.json |
[production] |
| 21:42 |
<pt1979@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1047.eqiad.wmnet with OS bullseye |
[production] |
| 21:38 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 21:33 |
<pt1979@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 21:13 |
<pt1979@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1047.eqiad.wmnet with OS bullseye |
[production] |
| 21:13 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 21:12 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 21:12 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 21:11 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 21:11 |
<inflatador> |
bking@cumin1001 restarting blazegraph on wdqs[1003-1013].eqiad.wmnet for T293862 |
[production] |
| 21:09 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 43385320f417052d8e60791b3cb970e6e3f088d5: fawiki: Set celebration logo for new vector (T304314; 2/2) (duration: 00m 53s) |
[production] |
| 21:07 |
<urbanecm@deploy1002> |
Synchronized static/images/mobile/copyright/wikipedia-fawiki-new-year.png: 43385320f417052d8e60791b3cb970e6e3f088d5: fawiki: Set celebration logo for new vector (T304314; 1/2) (duration: 00m 50s) |
[production] |
| 21:07 |
<thcipriani@deploy1002> |
Finished deploy [releng/phatality@15f8ec0]: Deploying phatality updates for opensearch 1.2.0 (duration: 00m 13s) |
[production] |
| 21:07 |
<thcipriani@deploy1002> |
Started deploy [releng/phatality@15f8ec0]: Deploying phatality updates for opensearch 1.2.0 |
[production] |
| 21:06 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 21:05 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 21:05 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 21:04 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 21:03 |
<urbanecm@deploy1002> |
Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 00m 50s) |
[production] |
| 20:44 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:43 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:43 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:43 |
<thcipriani@deploy1002> |
Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:773607|Start writing to $wmgAllServices the same value as to $wmfAllServices (T45956)]] (duration: 01m 17s) |
[production] |
| 20:42 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:32 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:32 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:31 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:31 |
<thcipriani@deploy1002> |
Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:768255|Stop writing to certain $wmf* global variables (T45956)]] (part 3) (duration: 00m 55s) |
[production] |
| 20:31 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:29 |
<thcipriani@deploy1002> |
Synchronized docroot/noc/db.php: Config: [[gerrit:768255|Stop writing to certain $wmf* global variables (T45956)]] (part II) (duration: 00m 51s) |
[production] |
| 20:28 |
<thcipriani@deploy1002> |
Synchronized tests: Config: [[gerrit:768255|Stop writing to certain $wmf* global variables (T45956)]] (part I) (duration: 00m 50s) |
[production] |
| 20:26 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:25 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:25 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:23 |
<thcipriani@deploy1002> |
Synchronized portals: Config: [[gerrit:773380|Bumping portals to master (T282012)]] (duration: 00m 52s) |
[production] |
| 20:22 |
<thcipriani@deploy1002> |
Synchronized portals/wikipedia.org/assets: Config: [[gerrit:773380|Bumping portals to master (T282012)]] (duration: 00m 52s) |
[production] |
| 20:21 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:19 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1145.eqiad.wmnet with reason: Maintenance |
[production] |