|
2023-10-05
ยง
|
| 21:07 |
<eevans@cumin1001> |
START - Cookbook sre.cassandra.roll-restart for nodes matching restbase2020.codfw.wmnet: Maybe cleanup leaked file descriptors(?) - eevans@cumin1001 |
[production] |
| 21:03 |
<thcipriani@deploy2002> |
Finished scap: Backport for [[gerrit:962082|[foundationwiki] Add Endowment, Agenda, Committee, and Memory namespaces (T347762 T347822 T348268)]], [[gerrit:963801|[foundationwiki] Provide 'translationadmin' group with 'edit-legal' right (T346187)]] (duration: 09m 56s) |
[production] |
| 20:58 |
<thcipriani@deploy2002> |
thcipriani and varnent: Continuing with sync |
[production] |
| 20:57 |
<eileen> |
civicrm upgraded from 05545fbc to 4329014b |
[production] |
| 20:55 |
<thcipriani@deploy2002> |
thcipriani and varnent: Backport for [[gerrit:962082|[foundationwiki] Add Endowment, Agenda, Committee, and Memory namespaces (T347762 T347822 T348268)]], [[gerrit:963801|[foundationwiki] Provide 'translationadmin' group with 'edit-legal' right (T346187)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 20:53 |
<thcipriani@deploy2002> |
Started scap: Backport for [[gerrit:962082|[foundationwiki] Add Endowment, Agenda, Committee, and Memory namespaces (T347762 T347822 T348268)]], [[gerrit:963801|[foundationwiki] Provide 'translationadmin' group with 'edit-legal' right (T346187)]] |
[production] |
| 20:49 |
<thcipriani@deploy2002> |
Finished scap: Backport for [[gerrit:963810|[Prototype] Add screen resolution to Typography prototype]], [[gerrit:963811|[Prototype] Edit project link page on reading prototype]] (duration: 23m 57s) |
[production] |
| 20:39 |
<thcipriani@deploy2002> |
jdrewniak and thcipriani: Continuing with sync |
[production] |
| 20:37 |
<thcipriani@deploy2002> |
jdrewniak and thcipriani: Backport for [[gerrit:963810|[Prototype] Add screen resolution to Typography prototype]], [[gerrit:963811|[Prototype] Edit project link page on reading prototype]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 20:25 |
<thcipriani@deploy2002> |
Started scap: Backport for [[gerrit:963810|[Prototype] Add screen resolution to Typography prototype]], [[gerrit:963811|[Prototype] Edit project link page on reading prototype]] |
[production] |
| 20:22 |
<thcipriani@deploy2002> |
Finished scap: Backport for [[gerrit:963799|Enable Minerva site notice for Nepali Wikipedia (newiki) (T347814)]] (duration: 08m 57s) |
[production] |
| 20:16 |
<thcipriani@deploy2002> |
ammarpad and thcipriani: Continuing with sync |
[production] |
| 20:14 |
<thcipriani@deploy2002> |
ammarpad and thcipriani: Backport for [[gerrit:963799|Enable Minerva site notice for Nepali Wikipedia (newiki) (T347814)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 20:14 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host an-master1003.eqiad.wmnet with OS bullseye |
[production] |
| 20:14 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host an-master1004.eqiad.wmnet with OS bullseye |
[production] |
| 20:13 |
<thcipriani@deploy2002> |
Started scap: Backport for [[gerrit:963799|Enable Minerva site notice for Nepali Wikipedia (newiki) (T347814)]] |
[production] |
| 18:51 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.reimage for host ganeti-test2004.codfw.wmnet with OS bullseye |
[production] |
| 18:47 |
<jhancock@cumin2002> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti-test2004'] |
[production] |
| 18:47 |
<jhancock@cumin2002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti-test2004'] |
[production] |
| 18:45 |
<jhancock@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 18:43 |
<jhancock@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 18:34 |
<jhuneidi@deploy2002> |
rebuilt and synchronized wikiversions files: group2 wikis to 1.41.0-wmf.29 refs T347080 |
[production] |
| 18:17 |
<sukhe> |
running authdns-update: T347054 |
[production] |
| 18:15 |
<jhuneidi@deploy2002> |
Synchronized php: group1 wikis to 1.41.0-wmf.29 refs T347080 (duration: 06m 12s) |
[production] |
| 18:08 |
<jhuneidi@deploy2002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.41.0-wmf.29 refs T347080 |
[production] |
| 17:39 |
<jhancock@cumin2002> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti-test2004'] |
[production] |
| 17:39 |
<jhancock@cumin2002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti-test2004'] |
[production] |
| 17:35 |
<jhancock@cumin2002> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti-test2004.mgmt.codfw.wmnet with reboot policy FORCED |
[production] |
| 17:30 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.provision for host ganeti-test2004.mgmt.codfw.wmnet with reboot policy FORCED |
[production] |
| 17:29 |
<jhancock@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti-test2004.mgmt.codfw.wmnet with reboot policy FORCED |
[production] |
| 17:27 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.provision for host ganeti-test2004.mgmt.codfw.wmnet with reboot policy FORCED |
[production] |
| 17:26 |
<jhancock@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 17:25 |
<jhancock@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:57 |
<bvibber> |
scaling back batch jobs for T312153 and T312152, will run these in further chunks as the new config rolls out |
[production] |
| 16:47 |
<bvibber> |
brion running requeueTranscodes.php on mwmaint2002 for VP9 transcode cleanup for T312153 |
[production] |
| 16:22 |
<volans> |
installed 7.3.1 on cumin1001 |
[production] |
| 16:19 |
<jbond@cumin2002> |
END (PASS) - Cookbook sre.puppetboard.restart-reboot (exit_code=0) rolling reboot on A:puppetboard |
[production] |
| 16:15 |
<jbond@cumin2002> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) puppetboard.discovery.wmnet. on all recursors |
[production] |
| 16:15 |
<jbond@cumin2002> |
START - Cookbook sre.dns.wipe-cache puppetboard.discovery.wmnet. on all recursors |
[production] |
| 16:12 |
<dcausse> |
cleaning up rdf-streaming-updater-staging swift bucket |
[production] |
| 16:11 |
<jbond@cumin2002> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) puppetboard.discovery.wmnet. on all recursors |
[production] |
| 16:10 |
<jbond@cumin2002> |
START - Cookbook sre.dns.wipe-cache puppetboard.discovery.wmnet. on all recursors |
[production] |
| 16:10 |
<jbond@cumin2002> |
START - Cookbook sre.puppetboard.restart-reboot rolling reboot on A:puppetboard |
[production] |
| 16:10 |
<jbond@cumin2002> |
END (ERROR) - Cookbook sre.puppet.renew-cert (exit_code=97) for sretest1001.eqiad.wmnet: Renew puppet certificate - jbond@cumin2002 |
[production] |
| 16:09 |
<jbond@cumin2002> |
START - Cookbook sre.puppet.renew-cert for sretest1001.eqiad.wmnet: Renew puppet certificate - jbond@cumin2002 |
[production] |
| 16:07 |
<cgoubert@deploy2002> |
Finished scap: Testing mw-on-k8s deployment for T348228 (duration: 02m 15s) |
[production] |
| 16:06 |
<jbond@cumin2002> |
END (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for sretest1001.eqiad.wmnet: Renew puppet certificate - jbond@cumin2002 |
[production] |
| 16:05 |
<cgoubert@deploy2002> |
Started scap: Testing mw-on-k8s deployment for T348228 |
[production] |
| 16:05 |
<jbond@cumin2002> |
END (PASS) - Cookbook sre.puppetboard.restart-reboot (exit_code=0) rolling reboot on A:puppetboard |
[production] |
| 16:05 |
<jbond@cumin2002> |
START - Cookbook sre.puppet.renew-cert for sretest1001.eqiad.wmnet: Renew puppet certificate - jbond@cumin2002 |
[production] |