|
2022-12-19
§
|
| 09:30 |
<aqu@deploy1002> |
Finished deploy [analytics/refinery@2d53aff] (thin): Fix bug fix in HDFS usage pipeline THIN [analytics/refinery@2d53aff] (duration: 00m 08s) |
[production] |
| 09:29 |
<aqu@deploy1002> |
Started deploy [analytics/refinery@2d53aff] (thin): Fix bug fix in HDFS usage pipeline THIN [analytics/refinery@2d53aff] |
[production] |
| 09:29 |
<aqu@deploy1002> |
Finished deploy [analytics/refinery@2d53aff]: Fix bug fix in HDFS usage pipeline [analytics/refinery@2d53aff] (duration: 08m 02s) |
[production] |
| 09:21 |
<aqu@deploy1002> |
Started deploy [analytics/refinery@2d53aff]: Fix bug fix in HDFS usage pipeline [analytics/refinery@2d53aff] |
[production] |
| 09:20 |
<aqu@deploy1002> |
Finished deploy [analytics/refinery@2d53aff] (hadoop-test): Fix bug fix in HDFS usage pipeline TEST [analytics/refinery@2d53aff] (duration: 01m 14s) |
[production] |
| 09:19 |
<aqu@deploy1002> |
Started deploy [analytics/refinery@2d53aff] (hadoop-test): Fix bug fix in HDFS usage pipeline TEST [analytics/refinery@2d53aff] |
[production] |
| 09:17 |
<aqu> |
About to deploy analytics/refinery (bug fix in HDFS usage pipeline) |
[production] |
| 09:15 |
<ladsgroup@deploy1002> |
Finished scap: Backport for [[gerrit:868867|Emergency: discussiontoolspageinfo return empty response in non-talk ns (T325477)]] (duration: 09m 24s) |
[production] |
| 09:14 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . |
[production] |
| 09:11 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . |
[production] |
| 09:07 |
<ladsgroup@deploy1002> |
ladsgroup and ladsgroup: Backport for [[gerrit:868867|Emergency: discussiontoolspageinfo return empty response in non-talk ns (T325477)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 09:06 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . |
[production] |
| 09:05 |
<ladsgroup@deploy1002> |
Started scap: Backport for [[gerrit:868867|Emergency: discussiontoolspageinfo return empty response in non-talk ns (T325477)]] |
[production] |
| 09:04 |
<dcausse> |
restarting blazegraph on wdqs1015 (BlazegraphFreeAllocatorsDecreasingRapidly) |
[production] |
| 09:03 |
<elukey@deploy1002> |
helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . |
[production] |
| 09:02 |
<elukey@deploy1002> |
helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . |
[production] |
| 09:01 |
<elukey@deploy1002> |
helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . |
[production] |
| 08:59 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . |
[production] |
| 08:58 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . |
[production] |
| 08:56 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . |
[production] |
| 08:37 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . |
[production] |
| 08:34 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . |
[production] |
| 08:33 |
<ayounsi@cumin1001> |
END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 29535 |
[production] |
| 08:32 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'email' for AS: 29535 |
[production] |
| 08:02 |
<moritzm> |
installing openexr security updates |
[production] |
| 07:22 |
<phedenskog@deploy1002> |
Finished deploy [performance/navtiming@5770d46]: (no justification provided) (duration: 00m 08s) |
[production] |
| 07:22 |
<phedenskog@deploy1002> |
Started deploy [performance/navtiming@5770d46]: (no justification provided) |
[production] |
| 07:05 |
<ayounsi@cumin1001> |
END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'configure' for AS: 28398 |
[production] |
| 07:05 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'configure' for AS: 28398 |
[production] |
| 07:04 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 11686 |
[production] |
| 06:48 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'configure' for AS: 11686 |
[production] |
| 03:08 |
<tstarling@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: disable wgDiscussionToolsABTest T325477 T321961 (duration: 15m 23s) |
[production] |
|
2022-12-18
§
|
| 19:40 |
<sukhe> |
ran sudo cumin -b 1 -s 30 'A:mw-api and A:eqiad' 'restart-php7.4-fpm' [at 18:55 UTC]: T325477 |
[production] |
| 19:31 |
<sukhe> |
[FINISHED] running sudo cumin -b 1 -s 30 'A:mw-api and A:eqiad' 'restart-php7.4-fpm' |
[production] |
| 19:31 |
<sukhe> |
[FINISHED] running sudo cumin -b 1 -s 30 'A:mw-api and A:eqiad' 'restart-php7.4-fpm' |
[production] |
| 18:55 |
<sukhe> |
running sudo cumin -b 1 -s 30 'A:mw-api and A:eqiad' 'restart-php7.4-fpm' |
[production] |
| 18:28 |
<cgoubert@cumin1001> |
END (FAIL) - Cookbook sre.mediawiki.restart-appservers (exit_code=99) |
[production] |
| 18:28 |
<cgoubert@cumin1001> |
START - Cookbook sre.mediawiki.restart-appservers |
[production] |
| 18:20 |
<cgoubert@cumin1001> |
END (FAIL) - Cookbook sre.mediawiki.restart-appservers (exit_code=99) |
[production] |
| 18:20 |
<cgoubert@cumin1001> |
START - Cookbook sre.mediawiki.restart-appservers |
[production] |
|
2022-12-16
§
|
| 19:55 |
<robh@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti4007.ulsfo.wmnet with OS bullseye |
[production] |
| 19:55 |
<robh@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - robh@cumin2002" |
[production] |
| 19:00 |
<robh@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - robh@cumin2002" |
[production] |
| 18:44 |
<robh@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti4007.ulsfo.wmnet with reason: host reimage |
[production] |
| 18:41 |
<robh@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti4007.ulsfo.wmnet with reason: host reimage |
[production] |
| 18:19 |
<robh@cumin2002> |
START - Cookbook sre.hosts.reimage for host ganeti4007.ulsfo.wmnet with OS bullseye |
[production] |
| 18:08 |
<robh@cumin2002> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti4007'] |
[production] |
| 17:56 |
<robh@cumin2002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti4007'] |
[production] |