3051-3100 of 10000 results (38ms)
2024-11-12 §
10:36 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply [production]
10:36 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
10:36 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
10:27 <wmbot~dcaro@urcuchillay> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice [tools]
10:20 <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice [tools]
10:12 <arnaudb@cumin1002> START - Cookbook sre.mysql.pool db2236 slowly with 10 steps - slow repool T373579 [production]
10:11 <wmbot~dcaro@urcuchillay> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice [tools]
10:08 <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice [tools]
09:59 <arnaudb@cumin1002> START - Cookbook sre.mysql.pool db2217 gradually with 4 steps - T379491 [production]
09:48 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2155 (T367781)', diff saved to https://phabricator.wikimedia.org/P71006 and previous config saved to /var/cache/conftool/dbconfig/20241112-094851-arnaudb.json [production]
09:41 <moritzm> update d-i netboot image for 12.8 point release T379600 [production]
09:39 <wmbot~dcaro@urcuchillay> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice [toolsbeta]
09:33 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P71005 and previous config saved to /var/cache/conftool/dbconfig/20241112-093343-arnaudb.json [production]
09:33 <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice [toolsbeta]
09:32 <wmbot~dcaro@urcuchillay> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice [toolsbeta]
09:31 <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice [toolsbeta]
09:31 <wmbot~dcaro@urcuchillay> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice [toolsbeta]
09:31 <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice [toolsbeta]
09:18 <urbanecm@deploy2002> Finished scap sync-world: Backport for [[gerrit:1090428|Revert "CirrusSearch: re-enable offloading weighted tags via EventBus"]] (duration: 06m 46s) [production]
09:18 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P71004 and previous config saved to /var/cache/conftool/dbconfig/20241112-091836-arnaudb.json [production]
09:17 <elukey@deploy2002> helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . [production]
09:14 <urbanecm@deploy2002> trainbranchbot, urbanecm: Continuing with sync [production]
09:14 <urbanecm@deploy2002> trainbranchbot, urbanecm: Backport for [[gerrit:1090428|Revert "CirrusSearch: re-enable offloading weighted tags via EventBus"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
09:11 <urbanecm@deploy2002> Started scap sync-world: Backport for [[gerrit:1090428|Revert "CirrusSearch: re-enable offloading weighted tags via EventBus"]] [production]
09:10 <urbanecm@deploy2002> Sync cancelled. [production]
09:03 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2155 (T367781)', diff saved to https://phabricator.wikimedia.org/P71002 and previous config saved to /var/cache/conftool/dbconfig/20241112-090329-arnaudb.json [production]
08:38 <urbanecm@deploy2002> pfischer, urbanecm: Backport for [[gerrit:1089826|CirrusSearch: re-enable offloading weighted tags via EventBus (T378983)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
08:36 <urbanecm@deploy2002> Started scap sync-world: Backport for [[gerrit:1089826|CirrusSearch: re-enable offloading weighted tags via EventBus (T378983)]] [production]
08:32 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1015.eqiad.wmnet [production]
08:31 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1015.eqiad.wmnet [production]
08:28 <urbanecm@deploy2002> Finished scap sync-world: Backport for [[gerrit:1089230|Fix WeightedTagsUpdater (T378664 T378983)]] (duration: 06m 59s) [production]
08:25 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1015.eqiad.wmnet [production]
08:21 <urbanecm@deploy2002> Started scap sync-world: Backport for [[gerrit:1089230|Fix WeightedTagsUpdater (T378664 T378983)]] [production]
08:19 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1009.eqiad.wmnet [production]
08:17 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1009.eqiad.wmnet [production]
08:04 <moritzm> installing apache security updates [production]
08:03 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2155 (T367781)', diff saved to https://phabricator.wikimedia.org/P71001 and previous config saved to /var/cache/conftool/dbconfig/20241112-080303-arnaudb.json [production]
08:02 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
08:02 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
08:02 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2155.codfw.wmnet with reason: Maintenance [production]
08:02 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db2155.codfw.wmnet with reason: Maintenance [production]
07:53 <jmm@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti-test2003 [production]
07:53 <jmm@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host ganeti-test2003 [production]
07:52 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on pc1017.eqiad.wmnet with reason: T378068, host is not pooled [production]
07:52 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 5 days, 0:00:00 on pc1017.eqiad.wmnet with reason: T378068, host is not pooled [production]
05:01 <mwpresync@deploy2002> Pruned MediaWiki: 1.43.0-wmf.28 (duration: 01m 52s) [production]
2024-11-11 §
23:20 <tgr|away> UTC late deploys done [production]
23:08 <tgr@deploy2002> scap failed: <CalledProcessError> Command '['sudo', '-u', 'mwbuilder', '-n', '--', '/usr/bin/scap', 'mwscript', '--no-local-config', '--directory', '/srv/mediawiki-staging', '--user', 'www-data', '--network', '--', 'purgeMessageBlobStore.php']' returned non-zero exit status 1. (scap version: 4.122.0) (duration: 11m 44s) [production]
23:02 <tgr@deploy2002> d3r1ck01, tgr: Continuing with sync [production]
22:59 <tgr@deploy2002> d3r1ck01, tgr: Backport for [[gerrit:1089807|PageUpdater: restore call to RevisionFromEditComplete (T379152)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]