|
2022-10-05
§
|
| 09:20 |
<dcausse> |
restarting blazegraph on wdqs1014 (BlazegraphFreeAllocatorsDecreasingRapidly) |
[production] |
| 09:15 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 09:11 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 09:11 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 09:10 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 09:09 |
<hoo@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Disable UnconnectedPagePagePropMigrationLegacyFormat for arwiki (duration: 03m 49s) |
[production] |
| 09:06 |
<moritzm> |
reimport ganeti 3.0.1-1~bpo10+1 to component/ganeti3 (got removed alongside via a reprepro bug/misfeature when the bullseye component was removed) |
[production] |
| 07:54 |
<elukey> |
restart kafka on kafka-logging1003 to pick up new PKI TLS settings |
[production] |
| 07:50 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:20:00 on kafka-logging1003.eqiad.wmnet with reason: Kafka PKI upgrade |
[production] |
| 07:49 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 0:20:00 on kafka-logging1003.eqiad.wmnet with reason: Kafka PKI upgrade |
[production] |
| 06:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2030 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35360 and previous config saved to /var/cache/conftool/dbconfig/20221005-065519-root.json |
[production] |
| 06:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2030 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35359 and previous config saved to /var/cache/conftool/dbconfig/20221005-064014-root.json |
[production] |
| 06:30 |
<elukey> |
restart kafka on kafka-logging1002 to pick up the new cert+settings for PKI |
[production] |
| 06:27 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:20:00 on kafka-logging1002.eqiad.wmnet with reason: Kafka PKI upgrade |
[production] |
| 06:27 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 0:20:00 on kafka-logging1002.eqiad.wmnet with reason: Kafka PKI upgrade |
[production] |
| 06:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2030 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35358 and previous config saved to /var/cache/conftool/dbconfig/20221005-062509-root.json |
[production] |
| 06:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2030 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35357 and previous config saved to /var/cache/conftool/dbconfig/20221005-061004-root.json |
[production] |
| 05:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2030 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35356 and previous config saved to /var/cache/conftool/dbconfig/20221005-055459-root.json |
[production] |
| 05:50 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 62044 |
[production] |
| 05:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2030 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35355 and previous config saved to /var/cache/conftool/dbconfig/20221005-053954-root.json |
[production] |
| 05:33 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'configure' for AS: 62044 |
[production] |
| 05:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2030 (re)pooling @ 3%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35354 and previous config saved to /var/cache/conftool/dbconfig/20221005-052449-root.json |
[production] |
| 05:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2030 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P35353 and previous config saved to /var/cache/conftool/dbconfig/20221005-050944-root.json |
[production] |
| 05:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es2030', diff saved to https://phabricator.wikimedia.org/P35352 and previous config saved to /var/cache/conftool/dbconfig/20221005-050018-root.json |
[production] |
| 02:59 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cloudvirt1023.eqiad.wmnet |
[production] |
| 02:21 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.dhcp for host cloudvirt1023.eqiad.wmnet |
[production] |
| 02:20 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cloudvirt1023.eqiad.wmnet |
[production] |
| 02:20 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.dhcp for host cloudvirt1023.eqiad.wmnet |
[production] |
| 02:19 |
<pt1979@cumin2002> |
END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host cloudvirt1023.eqiad.wmnet |
[production] |
| 02:19 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.dhcp for host cloudvirt1023.eqiad.wmnet |
[production] |
| 00:05 |
<sukhe> |
disable puppet on dns4003 till we resolve the puppet failures |
[production] |
|
2022-10-04
§
|
| 23:09 |
<andrew@cumin1001> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirt1023.eqiad.wmnet with OS bullseye |
[production] |
| 22:53 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1023.eqiad.wmnet with OS bullseye |
[production] |
| 21:28 |
<cjming> |
end of UTC late backport window |
[production] |
| 21:26 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 21:25 |
<cjming@deploy1002> |
Finished scap: Backport for [[gerrit:838210|Revert "Revert "Add wordmark and tagline for Bengali Wikibooks""]] (duration: 05m 06s) |
[production] |
| 21:25 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 21:25 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 21:24 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 21:21 |
<cjming@deploy1002> |
cjming and cjming: Backport for [[gerrit:838210|Revert "Revert "Add wordmark and tagline for Bengali Wikibooks""]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 21:20 |
<cjming@deploy1002> |
Started scap: Backport for [[gerrit:838210|Revert "Revert "Add wordmark and tagline for Bengali Wikibooks""]] |
[production] |
| 21:14 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 21:11 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 21:11 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 21:10 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 21:06 |
<cjming@deploy1002> |
Finished scap: Backport for [[gerrit:838101|Enable wgMinervaEnableSiteNotice for bnwikibooks (T319317)]] (duration: 05m 40s) |
[production] |
| 21:05 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 21:04 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 21:04 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 21:03 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |