|
2022-01-14
§
|
| 09:00 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM an-tool1008.eqiad.wmnet |
[production] |
| 08:58 |
<moritzm> |
rebooting an-tool1008 (running yarn.wikimedia.org) |
[production] |
| 08:58 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.reboot-vm for VM an-tool1008.eqiad.wmnet |
[production] |
| 08:57 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM an-tool1007.eqiad.wmnet |
[production] |
| 08:55 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.reboot-vm for VM an-tool1007.eqiad.wmnet |
[production] |
| 08:53 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM an-tool1005.eqiad.wmnet |
[production] |
| 08:51 |
<moritzm> |
rebooting an-tool1007 (running turnilo.wikimedia.org) |
[production] |
| 08:50 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.reboot-vm for VM an-tool1005.eqiad.wmnet |
[production] |
| 08:36 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM cuminunpriv1001.eqiad.wmnet |
[production] |
| 08:34 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.reboot-vm for VM cuminunpriv1001.eqiad.wmnet |
[production] |
| 08:33 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host pc2013.codfw.wmnet with OS bullseye |
[production] |
| 07:39 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc2012.codfw.wmnet with OS bullseye |
[production] |
| 07:05 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host pc2012.codfw.wmnet with OS bullseye |
[production] |
| 06:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove logpager group from s3 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P18735 and previous config saved to /var/cache/conftool/dbconfig/20220114-063554-marostegui.json |
[production] |
| 06:15 |
<marostegui> |
Failover m5 proxy from dbproxy1017 to dbproxy1021 T298586 |
[production] |
| 05:16 |
<legoktm> |
manually restarted discard_held_messages service on lists1001, failed with a spurious sqlalchemy issue about packets being out of order |
[production] |
| 00:36 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 00:32 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 00:32 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 00:30 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 00:25 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 00:24 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 00:24 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 00:23 |
<dduvall@deploy1002> |
rebuilt and synchronized wikiversions files: all wikis to 1.38.0-wmf.17 refs T293958 |
[production] |
| 00:20 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 00:15 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 00:15 |
<dduvall@deploy1002> |
Synchronized php: group1 wikis to 1.38.0-wmf.17 refs T293958 (duration: 01m 06s) |
[production] |
| 00:14 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 00:14 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 00:13 |
<dduvall@deploy1002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.38.0-wmf.17 refs T293958 |
[production] |
| 00:09 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 00:08 |
<dduvall@deploy1002> |
Synchronized php-1.38.0-wmf.17/includes/content/WikitextContentHandler.php: Backport: [[gerrit:753828|In WikitextContentHandler always use getFreshParser() (T299149)]] (duration: 01m 07s) |
[production] |
|
2022-01-13
§
|
| 22:40 |
<WFan> |
Updating payment-wiki, revision changed from 8497eae9 to 5cc9d5e0 |
[production] |
| 22:18 |
<dzahn@cumin1001> |
conftool action : set/pooled=false; selector: dnsdisc=miscweb |
[production] |
| 22:00 |
<dzahn@cumin1001> |
conftool action : set/pooled=true; selector: dnsdisc=miscweb |
[production] |
| 21:48 |
<mutante> |
running puppet on cp-ulsfo |
[production] |
| 21:47 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 21:44 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 21:44 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 21:42 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 21:02 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 20:55 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 20:55 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 20:48 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 20:43 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 20:37 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 20:37 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 20:31 |
<dduvall@deploy1002> |
rebuilt and synchronized wikiversions files: Revert "group1 wikis to 1.38.0-wmf.17" |
[production] |
| 20:31 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 20:28 |
<dduvall> |
rolling back wmf.17 from group1 due to a large increase in "Parser state cleared while parsing" across commons and group1 wikipedias (T293958, T299149) |
[production] |