|
2023-05-31
§
|
| 19:54 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dns1005.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 19:54 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dns1004.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 19:20 |
<JustHann_> |
we started swapping in dumpsdata1006 as primary nfs dumps server, replacing dumpsdata1005 at 16:55 UTC and completed at 19:09 UTC |
[production] |
| 19:19 |
<robh@cumin1001> |
START - Cookbook sre.hosts.provision for host dns1006.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 19:18 |
<robh@cumin1001> |
START - Cookbook sre.hosts.provision for host dns1005.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 19:17 |
<robh@cumin1001> |
START - Cookbook sre.hosts.provision for host dns1004.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 19:16 |
<robh@cumin1001> |
END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['dns1004'] |
[production] |
| 19:16 |
<robh@cumin1001> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dns1004'] |
[production] |
| 19:14 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['dns1006'] |
[production] |
| 19:14 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['dns1005'] |
[production] |
| 19:14 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['dns1004'] |
[production] |
| 19:13 |
<robh@cumin1001> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dns1006'] |
[production] |
| 19:13 |
<robh@cumin1001> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dns1005'] |
[production] |
| 19:13 |
<robh@cumin1001> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dns1004'] |
[production] |
| 19:12 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dns1006 |
[production] |
| 19:11 |
<robh@cumin1001> |
START - Cookbook sre.network.configure-switch-interfaces for host dns1006 |
[production] |
| 19:11 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dns1005 |
[production] |
| 19:10 |
<robh@cumin1001> |
START - Cookbook sre.network.configure-switch-interfaces for host dns1005 |
[production] |
| 19:09 |
<JustHann_> |
swapping in dumpsdata1006 as primary nfs dumps server, replacing dumpsdata1005 now completed! |
[production] |
| 18:59 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dns1004 |
[production] |
| 18:58 |
<robh@cumin1001> |
START - Cookbook sre.network.configure-switch-interfaces for host dns1004 |
[production] |
| 18:57 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 18:57 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: new dns100[345] - robh@cumin1001" |
[production] |
| 18:56 |
<robh@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: new dns100[345] - robh@cumin1001" |
[production] |
| 18:53 |
<robh@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 18:40 |
<dduvall@deploy1002> |
Synchronized php: group1 wikis to 1.41.0-wmf.11 refs T337525 (duration: 06m 02s) |
[production] |
| 18:38 |
<mforns@deploy1002> |
Finished deploy [airflow-dags/analytics_product@b3eb622]: (no justification provided) (duration: 00m 07s) |
[production] |
| 18:38 |
<mforns@deploy1002> |
Started deploy [airflow-dags/analytics_product@b3eb622]: (no justification provided) |
[production] |
| 18:34 |
<dduvall@deploy1002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.41.0-wmf.11 refs T337525 |
[production] |
| 18:01 |
<TheresNoTime> |
ran `wikiadmin2023@10.64.32.139(huwiki)> UPDATE thread SET thread_signature = '<span title="bétaverzió"> <!--<font style="text-decoration: blink;">--><font color="red">♥</font><font color="white">♥</font><font color="green">♥</font> </font> [[User:Gubbubu|<font color="green" face="Lucida calligraphy">Γουββος Θιλο' WHERE thread_id = 1288;` (with `BEGIN`/`COMMIT`) for T337700 |
[production] |
| 17:31 |
<ladsgroup@deploy1002> |
Backport cancelled. |
[production] |
| 17:30 |
<ladsgroup@deploy1002> |
Finished scap: Backport for [[gerrit:924885|Remove legacy encoding option from dawiktionary (T128155)]] (duration: 12m 54s) |
[production] |
| 17:18 |
<ladsgroup@deploy1002> |
ladsgroup: Backport for [[gerrit:924885|Remove legacy encoding option from dawiktionary (T128155)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet |
[production] |
| 17:17 |
<ladsgroup@deploy1002> |
Started scap: Backport for [[gerrit:924885|Remove legacy encoding option from dawiktionary (T128155)]] |
[production] |
| 17:10 |
<brett@deploy1002> |
Unlocked for deployment [ALL REPOSITORIES]: LVS maintenance in codfw, blocking deploys T322937 (duration: 11m 07s) |
[production] |
| 17:10 |
<brett> |
Maglev LVS scheduler rollout in codfw finished - T263797 |
[production] |
| 17:10 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.cdn.run-puppet-restart-varnish (exit_code=0) rolling custom on P{cp[2037,2039,2041].codfw.wmnet} and A:cp |
[production] |
| 16:59 |
<brett@deploy1002> |
Locking from deployment [ALL REPOSITORIES]: LVS maintenance in codfw, blocking deploys T322937 |
[production] |
| 16:37 |
<mfossati@deploy1002> |
Finished deploy [airflow-dags/platform_eng@5379d83]: (no justification provided) (duration: 00m 34s) |
[production] |
| 16:37 |
<mfossati@deploy1002> |
Started deploy [airflow-dags/platform_eng@5379d83]: (no justification provided) |
[production] |
| 16:22 |
<elukey> |
`systemctl reset-failed session-c6111.scope session-c7230.scope` on stat1005 to clear old alerts |
[production] |
| 16:20 |
<vgutierrez@cumin1001> |
START - Cookbook sre.cdn.run-puppet-restart-varnish rolling custom on P{cp[2037,2039,2041].codfw.wmnet} and A:cp |
[production] |
| 16:13 |
<vgutierrez> |
repool cp2035 - T337247 T323557 |
[production] |
| 16:12 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp2035.codfw.wmnet |
[production] |
| 16:12 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.remove-downtime for cp2035.codfw.wmnet |
[production] |
| 16:10 |
<otto@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 16:10 |
<otto@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 16:08 |
<otto@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 16:08 |
<otto@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 16:04 |
<otto@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |