2801-2850 of 10000 results (113ms)
2024-10-21 ยง
18:52 <swfrench@cumin2002> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-low-traffic-eqiad (T377040) [production]
18:51 <zabe@deploy2002> Finished scap sync-world: Backport for [[gerrit:1081463|s4: Reduce revision-slots cache expiry to 60 seconds (T183490)]] (duration: 16m 09s) [production]
18:51 <ejegg> fundraising civicrm upgraded from cfb0def0 to 36660cb3 [production]
18:45 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host backup1012.eqiad.wmnet with OS bookworm [production]
18:45 <elukey@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1002" [production]
18:44 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P70418 and previous config saved to /var/cache/conftool/dbconfig/20241021-184424-ladsgroup.json [production]
18:43 <swfrench@cumin2002> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-secondary-eqiad (T377040) [production]
18:42 <zabe@deploy2002> zabe: Continuing with sync [production]
18:42 <zabe@deploy2002> zabe: Backport for [[gerrit:1081463|s4: Reduce revision-slots cache expiry to 60 seconds (T183490)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
18:37 <elukey@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1002" [production]
18:37 <swfrench@cumin2002> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-secondary-eqiad (T377040) [production]
18:36 <swfrench-wmf> ran and enabled puppet agent on 'A:lvs and A:eqiad' - T377040 [production]
18:35 <zabe@deploy2002> Started scap sync-world: Backport for [[gerrit:1081463|s4: Reduce revision-slots cache expiry to 60 seconds (T183490)]] [production]
18:32 <swfrench-wmf> ran disable-puppet on 'A:lvs and (A:eqiad or A:codfw)' - T377040 [production]
18:29 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P70417 and previous config saved to /var/cache/conftool/dbconfig/20241021-182916-ladsgroup.json [production]
18:23 <swfrench@cumin2002> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-low-traffic-codfw (T377040) [production]
18:22 <swfrench@cumin2002> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-low-traffic-codfw (T377040) [production]
18:20 <swfrench@cumin2002> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-secondary-codfw (T377040) [production]
18:19 <swfrench@cumin2002> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-secondary-codfw (T377040) [production]
18:19 <swfrench-wmf> ran and enabled pupppet agent on 'A:lvs and A:codfw' - T377040 [production]
18:15 <swfrench@cumin2002> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-low-traffic-eqiad (T377040) [production]
18:14 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup1012.eqiad.wmnet with reason: host reimage [production]
18:14 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2158 (T376905)', diff saved to https://phabricator.wikimedia.org/P70416 and previous config saved to /var/cache/conftool/dbconfig/20241021-181410-ladsgroup.json [production]
18:11 <elukey@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on backup1012.eqiad.wmnet with reason: host reimage [production]
18:09 <swfrench@cumin2002> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-low-traffic-eqiad (T377040) [production]
18:09 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2158 (T376905)', diff saved to https://phabricator.wikimedia.org/P70415 and previous config saved to /var/cache/conftool/dbconfig/20241021-180654-ladsgroup.json [production]
18:09 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
18:08 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
18:07 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2158.codfw.wmnet with reason: Maintenance [production]
18:07 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2158.codfw.wmnet with reason: Maintenance [production]
18:06 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2151 (T376905)', diff saved to https://phabricator.wikimedia.org/P70414 and previous config saved to /var/cache/conftool/dbconfig/20241021-180612-ladsgroup.json [production]
18:06 <swfrench@cumin2002> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-secondary-eqiad (T377040) [production]
18:05 <swfrench@cumin2002> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-secondary-eqiad (T377040) [production]
18:04 <swfrench-wmf> ran and enabled pupppet agent on 'A:lvs and A:eqiad' - T377040 [production]
17:59 <swfrench-wmf> ran disable-puppet on 'A:lvs and (A:eqiad or A:codfw)' - T377040 [production]
17:56 <elukey@cumin1002> START - Cookbook sre.hosts.reimage for host backup1012.eqiad.wmnet with OS bookworm [production]
17:53 <elukey@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host backup1012.eqiad.wmnet with OS bookworm [production]
17:53 <elukey@cumin1002> START - Cookbook sre.hosts.reimage for host backup1012.eqiad.wmnet with OS bookworm [production]
17:52 <dduvall@deploy2002> Installing scap version "4.115.0" for 209 hosts [production]
17:51 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P70413 and previous config saved to /var/cache/conftool/dbconfig/20241021-175105-ladsgroup.json [production]
17:50 <xcollazo@deploy2002> Finished deploy [airflow-dags/analytics@671896c]: Deploy T375402. (duration: 01m 04s) [production]
17:48 <xcollazo@deploy2002> Started deploy [airflow-dags/analytics@671896c]: Deploy T375402. [production]
17:44 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host backup1012.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
17:43 <elukey@cumin1002> START - Cookbook sre.hosts.provision for host backup1012.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
17:42 <elukey@cumin1002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host backup1012.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
17:41 <elukey@cumin1002> START - Cookbook sre.hosts.provision for host backup1012.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
17:35 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P70412 and previous config saved to /var/cache/conftool/dbconfig/20241021-173558-ladsgroup.json [production]
17:20 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2151 (T376905)', diff saved to https://phabricator.wikimedia.org/P70411 and previous config saved to /var/cache/conftool/dbconfig/20241021-172051-ladsgroup.json [production]
17:12 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2151 (T376905)', diff saved to https://phabricator.wikimedia.org/P70410 and previous config saved to /var/cache/conftool/dbconfig/20241021-171138-ladsgroup.json [production]
17:11 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2151.codfw.wmnet with reason: Maintenance [production]