501-550 of 10000 results (79ms)
2024-10-30 ยง
12:55 <andrewtavis-wmde@deploy2002> Finished deploy [airflow-dags/wmde@ec4746b]: (no justification provided) (duration: 00m 11s) [production]
12:54 <andrewtavis-wmde@deploy2002> Started deploy [airflow-dags/wmde@ec4746b]: (no justification provided) [production]
12:51 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2156 (T376905)', diff saved to https://phabricator.wikimedia.org/P70702 and previous config saved to /var/cache/conftool/dbconfig/20241030-125150-ladsgroup.json [production]
12:43 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2156 (T376905)', diff saved to https://phabricator.wikimedia.org/P70701 and previous config saved to /var/cache/conftool/dbconfig/20241030-124316-ladsgroup.json [production]
12:43 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance [production]
12:43 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance [production]
12:43 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2156.codfw.wmnet with reason: Maintenance [production]
12:43 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2156.codfw.wmnet with reason: Maintenance [production]
12:42 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2149 (T376905)', diff saved to https://phabricator.wikimedia.org/P70700 and previous config saved to /var/cache/conftool/dbconfig/20241030-124256-ladsgroup.json [production]
12:30 <dreamyjazz@deploy2002> Finished scap sync-world: Backport for [[gerrit:1084743|Handle a missing parent block in GlobalBlockLookup::getUserBlock (T378447)]], [[gerrit:1084742|Handle a missing parent block in GlobalBlockLookup::getUserBlock (T378447)]], [[gerrit:1084748|globalblocks API: Hide autoblocks when target param has username and IP (T377855)]] (duration: 10m 28s) [production]
12:27 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P70699 and previous config saved to /var/cache/conftool/dbconfig/20241030-122749-ladsgroup.json [production]
12:25 <dreamyjazz@deploy2002> dreamyjazz: Continuing with sync [production]
12:22 <mvolz@deploy2002> helmfile [eqiad] DONE helmfile.d/services/citoid: apply [production]
12:22 <dreamyjazz@deploy2002> dreamyjazz: Backport for [[gerrit:1084743|Handle a missing parent block in GlobalBlockLookup::getUserBlock (T378447)]], [[gerrit:1084742|Handle a missing parent block in GlobalBlockLookup::getUserBlock (T378447)]], [[gerrit:1084748|globalblocks API: Hide autoblocks when target param has username and IP (T377855)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
12:22 <mvolz@deploy2002> helmfile [eqiad] START helmfile.d/services/citoid: apply [production]
12:21 <ayounsi@cumin1002> END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox [production]
12:21 <ayounsi@cumin1002> START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox [production]
12:20 <dreamyjazz@deploy2002> Started scap sync-world: Backport for [[gerrit:1084743|Handle a missing parent block in GlobalBlockLookup::getUserBlock (T378447)]], [[gerrit:1084742|Handle a missing parent block in GlobalBlockLookup::getUserBlock (T378447)]], [[gerrit:1084748|globalblocks API: Hide autoblocks when target param has username and IP (T377855)]] [production]
12:19 <mvolz@deploy2002> helmfile [codfw] DONE helmfile.d/services/citoid: apply [production]
12:19 <mvolz@deploy2002> helmfile [codfw] START helmfile.d/services/citoid: apply [production]
12:18 <ayounsi@cumin1002> END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary [production]
12:17 <ayounsi@cumin1002> START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary [production]
12:17 <mvolz@deploy2002> helmfile [staging] DONE helmfile.d/services/citoid: apply [production]
12:16 <mvolz@deploy2002> helmfile [staging] START helmfile.d/services/citoid: apply [production]
12:12 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P70698 and previous config saved to /var/cache/conftool/dbconfig/20241030-121242-ladsgroup.json [production]
12:12 <moritzm> installing podman security updates [production]
12:11 <joal@deploy2002> Finished deploy [analytics/refinery@0855ce2] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@0855ce28] (duration: 03m 41s) [production]
12:07 <joal@deploy2002> Started deploy [analytics/refinery@0855ce2] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@0855ce28] [production]
12:04 <joal@deploy2002> Finished deploy [analytics/refinery@0855ce2] (thin): Regular analytics weekly train THIN [analytics/refinery@0855ce28] (duration: 06m 54s) [production]
11:57 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2149 (T376905)', diff saved to https://phabricator.wikimedia.org/P70697 and previous config saved to /var/cache/conftool/dbconfig/20241030-115735-ladsgroup.json [production]
11:57 <joal@deploy2002> Started deploy [analytics/refinery@0855ce2] (thin): Regular analytics weekly train THIN [analytics/refinery@0855ce28] [production]
11:55 <joal@deploy2002> Finished deploy [analytics/refinery@0855ce2]: Regular analytics weekly train [analytics/refinery@0855ce28] (duration: 08m 14s) [production]
11:48 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2149 (T376905)', diff saved to https://phabricator.wikimedia.org/P70696 and previous config saved to /var/cache/conftool/dbconfig/20241030-114808-ladsgroup.json [production]
11:48 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2149.codfw.wmnet with reason: Maintenance [production]
11:47 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2149.codfw.wmnet with reason: Maintenance [production]
11:47 <joal@deploy2002> Started deploy [analytics/refinery@0855ce2]: Regular analytics weekly train [analytics/refinery@0855ce28] [production]
11:44 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 12:00:00 on db1216.eqiad.wmnet with reason: Maintenance [production]
11:43 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 2 days, 12:00:00 on db1216.eqiad.wmnet with reason: Maintenance [production]
11:41 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2139.codfw.wmnet with reason: Maintenance [production]
11:41 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2139.codfw.wmnet with reason: Maintenance [production]
11:39 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2016.codfw.wmnet [production]
11:39 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ml-etcd2003.codfw.wmnet to plain [production]
11:38 <jmm@cumin2002> START - Cookbook sre.ganeti.changedisk for changing disk type of ml-etcd2003.codfw.wmnet to plain [production]
11:38 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1011.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [production]
11:38 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2016.codfw.wmnet [production]
11:37 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2016.codfw.wmnet [production]
11:36 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ml-etcd2003.codfw.wmnet to drbd [production]
11:33 <elukey@cumin1002> START - Cookbook sre.hosts.provision for host ml-serve1011.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [production]
11:28 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1010.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [production]
11:26 <jmm@cumin2002> START - Cookbook sre.ganeti.changedisk for changing disk type of ml-etcd2003.codfw.wmnet to drbd [production]