|
2025-04-15
§
|
| 09:43 |
<dcausse@deploy1003> |
Started deploy [wdqs/wdqs@fe88851]: version 0.3.156 (T326311) |
[production] |
| 09:28 |
<arturo> |
added `toolsbeta-tofu` bot account with `member` permissions T391474 |
[toolsbeta] |
| 09:15 |
<jnuche@deploy1003> |
sync-world aborted: testwikis to 1.44.0-wmf.25 refs T386220 (duration: 14m 36s) |
[production] |
| 09:00 |
<jnuche@deploy1003> |
Started scap sync-world: testwikis to 1.44.0-wmf.25 refs T386220 |
[production] |
| 08:51 |
<dcausse@deploy1003> |
Finished deploy [wdqs/wdqs@4186ae7] (wcqs): test deploy new scap config to wcqs2001.codfw.wmnet (T221709) (duration: 00m 20s) |
[production] |
| 08:51 |
<dcausse@deploy1003> |
Started deploy [wdqs/wdqs@4186ae7] (wcqs): test deploy new scap config to wcqs2001.codfw.wmnet (T221709) |
[production] |
| 08:42 |
<XioNoX> |
drain arelion eqsin-codfw link |
[production] |
| 08:09 |
<dcausse@deploy1003> |
Finished deploy [wdqs/wdqs@4186ae7]: test deploy new scap config to wdqs2025.codfw.wmnet (T221709) (duration: 00m 18s) |
[production] |
| 08:09 |
<brouberol@deploy1003> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply |
[production] |
| 08:09 |
<dcausse@deploy1003> |
Started deploy [wdqs/wdqs@4186ae7]: test deploy new scap config to wdqs2025.codfw.wmnet (T221709) |
[production] |
| 08:08 |
<brouberol@deploy1003> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply |
[production] |
| 07:47 |
<godog> |
upgrade thanos to 0.38.0 on prometheus100[57] - T383966 |
[production] |
| 07:28 |
<Emperor> |
make sure all disks are mounted correctly prior to disk-swap testing T391854 ms-be1091 |
[production] |
| 07:28 |
<Emperor> |
make sure all disks are mounted correctly prior to disk-swap testing T391854 |
[production] |
| 07:10 |
<elukey@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ms-be1091.eqiad.wmnet with reason: dcops maintenance |
[production] |
| 07:06 |
<vgutierrez@cumin1002> |
START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_codfw |
[production] |
| 07:06 |
<vgutierrez@cumin1002> |
START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_codfw |
[production] |
| 07:06 |
<vgutierrez@cumin1002> |
START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_eqsin |
[production] |
| 07:05 |
<kartik@deploy1003> |
helmfile [staging] DONE helmfile.d/services/machinetranslation: apply |
[production] |
| 07:05 |
<vgutierrez@cumin1002> |
START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_eqsin |
[production] |
| 07:04 |
<vgutierrez> |
rolling upgrade to varnish 7.1.1-1.1~bpo11+wmf3 in eqsin and codfw - T391334 |
[production] |
| 06:50 |
<kartik@deploy1003> |
helmfile [staging] START helmfile.d/services/machinetranslation: apply |
[production] |
| 06:48 |
<kart_> |
Updated cxserver to 2025-04-07-053106-production (T390732, T390711) |
[production] |
| 06:48 |
<kartik@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/cxserver: apply |
[production] |
| 06:47 |
<kartik@deploy1003> |
helmfile [eqiad] START helmfile.d/services/cxserver: apply |
[production] |
| 06:46 |
<kartik@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/cxserver: apply |
[production] |
| 06:45 |
<kartik@deploy1003> |
helmfile [codfw] START helmfile.d/services/cxserver: apply |
[production] |
| 06:45 |
<kartik@deploy1003> |
helmfile [staging] DONE helmfile.d/services/cxserver: apply |
[production] |
| 06:44 |
<kartik@deploy1003> |
helmfile [staging] START helmfile.d/services/cxserver: apply |
[production] |
| 05:03 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repool pc6 T391454', diff saved to https://phabricator.wikimedia.org/P75003 and previous config saved to /var/cache/conftool/dbconfig/20250415-050307-marostegui.json |
[production] |
| 04:57 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2016.codfw.wmnet,pc1016.eqiad.wmnet with reason: Maintenance |
[production] |
| 04:57 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool pc6 T391454', diff saved to https://phabricator.wikimedia.org/P75002 and previous config saved to /var/cache/conftool/dbconfig/20250415-045700-marostegui.json |
[production] |
| 04:10 |
<mwpresync@deploy1003> |
Pruned MediaWiki: 1.44.0-wmf.22 (duration: 10m 03s) |
[production] |
| 03:43 |
<mwpresync@deploy1003> |
sync-world failed: <CalledProcessError> Command 'sudo -u mwbuilder /srv/mwbuilder/release/make-container-image/build-images.py /srv/mediawiki-staging/scap/image-build --staging-dir /srv/mediawiki-staging --mediawiki-versions 1.44.0-wmf.24,1.44.0-wmf.25 --multiversion-image-name docker-registry.discovery.wmnet/restricted/mediawiki-multiversion --multiversion-debug-image-name docker-registry.discov |
[production] |
| 03:02 |
<mwpresync@deploy1003> |
Started scap sync-world: testwikis to 1.44.0-wmf.25 refs T386220 |
[production] |
| 02:32 |
<ejegg> |
payments-wiki upgraded from ef9284aa to ba6e8d65 |
[production] |
| 02:06 |
<jhancock@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1181.eqiad.wmnet with OS bullseye |
[production] |
| 01:32 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.reimage for host an-worker1181.eqiad.wmnet with OS bullseye |
[production] |
| 01:31 |
<jhancock@cumin2002> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-worker1181'] |
[production] |
| 01:30 |
<jhancock@cumin2002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-worker1181'] |
[production] |
| 01:24 |
<jhancock@cumin2002> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-worker1181.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL |
[production] |
| 01:03 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.provision for host an-worker1181.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL |
[production] |