101-150 of 10000 results (113ms)
2024-10-07 §
09:27 <arnaudb@cumin1002> dbctl commit (dc=all): 'missing commit', diff saved to https://phabricator.wikimedia.org/P69476 and previous config saved to /var/cache/conftool/dbconfig/20241007-092714-arnaudb.json [production]
09:19 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1246 (re)pooling @ 1%: T374215', diff saved to https://phabricator.wikimedia.org/P69474 and previous config saved to /var/cache/conftool/dbconfig/20241007-091953-arnaudb.json [production]
09:18 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1233 (re)pooling @ 1%: T374215', diff saved to https://phabricator.wikimedia.org/P69473 and previous config saved to /var/cache/conftool/dbconfig/20241007-091854-arnaudb.json [production]
09:13 <arnaudb@cumin1002> END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1233.eqiad.wmnet onto db1246.eqiad.wmnet [production]
08:37 <aqu@deploy2002> Finished deploy [airflow-dags/analytics@1699d34]: Refine staging fixes [airflow-dags@1699d34f] (duration: 04m 43s) [production]
08:32 <aqu@deploy2002> Started deploy [airflow-dags/analytics@1699d34]: Refine staging fixes [airflow-dags@1699d34f] [production]
08:24 <aqu@deploy2002> Finished deploy [airflow-dags/analytics_test@4b69f50]: Stage Refine fixes on test cluster [airflow-dags@4b69f503] (duration: 00m 13s) [production]
08:24 <aqu@deploy2002> Started deploy [airflow-dags/analytics_test@4b69f50]: Stage Refine fixes on test cluster [airflow-dags@4b69f503] [production]
08:02 <aqu@deploy2002> Finished deploy [airflow-dags/analytics_test@4b69f50]: Stage Refine fixes on test cluster [airflow-dags@4b69f503] (duration: 00m 18s) [production]
08:02 <jelto@deploy2002> helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply [production]
08:02 <aqu@deploy2002> Started deploy [airflow-dags/analytics_test@4b69f50]: Stage Refine fixes on test cluster [airflow-dags@4b69f503] [production]
08:02 <jelto@deploy2002> helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply [production]
08:01 <jelto@deploy2002> helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply [production]
08:01 <jelto@deploy2002> helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply [production]
08:00 <aborrero@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudgw1002.eqiad.wmnet [production]
07:57 <aborrero@cumin1002> START - Cookbook sre.hosts.reboot-single for host cloudgw1002.eqiad.wmnet [production]
07:56 <arnaudb@cumin1002> START - Cookbook sre.mysql.clone of db1233.eqiad.wmnet onto db1246.eqiad.wmnet [production]
07:56 <arnaudb@cumin1002> dbctl commit (dc=all): 'T374215 db1233 depool as clone source for db1246', diff saved to https://phabricator.wikimedia.org/P69471 and previous config saved to /var/cache/conftool/dbconfig/20241007-075611-arnaudb.json [production]
07:56 <hashar> UTC morning backport window completed [production]
07:54 <hashar@deploy2002> Finished scap sync-world: Backport for [[gerrit:1077392|logos: Sync config.yaml and logos.php (T374430)]], [[gerrit:1077400|hawiki: Add temporary logo (T376049)]] (duration: 11m 19s) [production]
07:49 <hashar@deploy2002> ammarpad, hashar: Continuing with sync [production]
07:45 <hashar@deploy2002> ammarpad, hashar: Backport for [[gerrit:1077392|logos: Sync config.yaml and logos.php (T374430)]], [[gerrit:1077400|hawiki: Add temporary logo (T376049)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
07:43 <hashar@deploy2002> Started scap sync-world: Backport for [[gerrit:1077392|logos: Sync config.yaml and logos.php (T374430)]], [[gerrit:1077400|hawiki: Add temporary logo (T376049)]] [production]
07:42 <hashar@deploy2002> Finished scap sync-world: Backport for [[gerrit:1077422|Revert "wikimaniawiki: Update logos to 2024"]] (duration: 21m 40s) [production]
07:04 <jelto@deploy2002> helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply [production]
07:04 <ayounsi@cumin1002> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 64315 [production]
07:04 <ayounsi@cumin1002> START - Cookbook sre.network.peering with action 'email' for AS: 64315 [production]
07:04 <jelto@deploy2002> helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply [production]
2024-10-06 §
22:59 <eileen> civicrm upgraded from 45855ff4 to f2095695 [production]
2024-10-05 §
19:43 <isaranto@deploy2002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
16:45 <hnowlan@deploy1003> helmfile [codfw] DONE helmfile.d/services/thumbor: apply [production]
16:41 <hnowlan@deploy1003> helmfile [codfw] START helmfile.d/services/thumbor: apply [production]
16:40 <hnowlan@deploy1003> helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [production]
16:37 <hnowlan@deploy1003> helmfile [eqiad] START helmfile.d/services/thumbor: apply [production]
16:36 <hnowlan@deploy1003> helmfile [staging] DONE helmfile.d/services/thumbor: apply [production]
16:36 <hnowlan@deploy1003> helmfile [staging] START helmfile.d/services/thumbor: apply [production]
13:31 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1192 (T367856)', diff saved to https://phabricator.wikimedia.org/P69470 and previous config saved to /var/cache/conftool/dbconfig/20241005-133058-ladsgroup.json [production]
13:30 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 12:00:00 on db1192.eqiad.wmnet with reason: Maintenance [production]
13:30 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 2 days, 12:00:00 on db1192.eqiad.wmnet with reason: Maintenance [production]
13:30 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1178 (T367856)', diff saved to https://phabricator.wikimedia.org/P69469 and previous config saved to /var/cache/conftool/dbconfig/20241005-133036-ladsgroup.json [production]
13:15 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P69468 and previous config saved to /var/cache/conftool/dbconfig/20241005-131529-ladsgroup.json [production]
13:00 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P69467 and previous config saved to /var/cache/conftool/dbconfig/20241005-130022-ladsgroup.json [production]
12:45 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1178 (T367856)', diff saved to https://phabricator.wikimedia.org/P69466 and previous config saved to /var/cache/conftool/dbconfig/20241005-124515-ladsgroup.json [production]
2024-10-04 §
17:48 <ejegg> fundraising civicrm upgraded from 90199f62 to 45855ff4 [production]
16:21 <ayounsi@cumin1002> END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host sretest2001.codfw.wmnet [production]
16:00 <bking@cumin2002> END (FAIL) - Cookbook sre.wdqs.categories-reload (exit_code=99) reloading categories to wdqs-categories1001.eqiad.wmnet [production]
14:29 <mforns@deploy2002> Finished deploy [airflow-dags/analytics@4b69f50]: add category to commons impact metrics allowlist (duration: 01m 48s) [production]
14:28 <mforns@deploy2002> Started deploy [airflow-dags/analytics@4b69f50]: add category to commons impact metrics allowlist [production]
13:54 <bking@cumin2002> START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs-categories1001.eqiad.wmnet [production]
13:33 <bking@cumin2002> END (ERROR) - Cookbook sre.wdqs.categories-reload (exit_code=97) reloading categories to wdqs-categories1001.eqiad.wmnet [production]