1801-1850 of 10000 results (83ms)
2023-08-29 ยง
20:55 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P51953 and previous config saved to /var/cache/conftool/dbconfig/20230829-205546-ladsgroup.json [production]
20:40 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2155 (T343718)', diff saved to https://phabricator.wikimedia.org/P51952 and previous config saved to /var/cache/conftool/dbconfig/20230829-204039-ladsgroup.json [production]
20:16 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:953315|clienthints: Raise maxlag for API back to default for group0 and 1 (T344797)]] (duration: 07m 13s) [production]
20:10 <urbanecm@deploy1002> urbanecm and dreamyjazz: Continuing with sync [production]
20:10 <urbanecm@deploy1002> urbanecm and dreamyjazz: Backport for [[gerrit:953315|clienthints: Raise maxlag for API back to default for group0 and 1 (T344797)]] synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
20:09 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:953315|clienthints: Raise maxlag for API back to default for group0 and 1 (T344797)]] [production]
19:52 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1190 (T343718)', diff saved to https://phabricator.wikimedia.org/P51951 and previous config saved to /var/cache/conftool/dbconfig/20230829-195215-ladsgroup.json [production]
19:52 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance [production]
19:52 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance [production]
19:51 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1160 (T343718)', diff saved to https://phabricator.wikimedia.org/P51950 and previous config saved to /var/cache/conftool/dbconfig/20230829-195154-ladsgroup.json [production]
19:36 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P51949 and previous config saved to /var/cache/conftool/dbconfig/20230829-193648-ladsgroup.json [production]
19:35 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase1033.eqiad.wmnet [production]
19:32 <ayounsi@cumin1001> END (ERROR) - Cookbook sre.network.tls (exit_code=97) for network device asw2-c2-eqiad [production]
19:32 <ayounsi@cumin1001> START - Cookbook sre.network.tls for network device asw2-c2-eqiad [production]
19:32 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-f1-eqiad [production]
19:30 <ayounsi@cumin1001> START - Cookbook sre.network.tls for network device ssw1-f1-eqiad [production]
19:30 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-e1-eqiad [production]
19:27 <ayounsi@cumin1001> START - Cookbook sre.network.tls for network device ssw1-e1-eqiad [production]
19:27 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f3-eqiad [production]
19:26 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host restbase1033.eqiad.wmnet [production]
19:25 <ayounsi@cumin1001> START - Cookbook sre.network.tls for network device lsw1-f3-eqiad [production]
19:25 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f2-eqiad [production]
19:24 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudservices1006.eqiad.wmnet with OS bullseye [production]
19:24 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
19:23 <ayounsi@cumin1001> START - Cookbook sre.network.tls for network device lsw1-f2-eqiad [production]
19:23 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f1-eqiad [production]
19:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P51948 and previous config saved to /var/cache/conftool/dbconfig/20230829-192141-ladsgroup.json [production]
19:20 <ayounsi@cumin1001> START - Cookbook sre.network.tls for network device lsw1-f1-eqiad [production]
19:20 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e3-eqiad [production]
19:18 <ayounsi@cumin1001> START - Cookbook sre.network.tls for network device lsw1-e3-eqiad [production]
19:18 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e2-eqiad [production]
19:18 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase1026.eqiad.wmnet [production]
19:18 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
19:16 <ayounsi@cumin1001> START - Cookbook sre.network.tls for network device lsw1-e2-eqiad [production]
19:16 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e1-eqiad [production]
19:13 <ayounsi@cumin1001> START - Cookbook sre.network.tls for network device lsw1-e1-eqiad [production]
19:11 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.debug (exit_code=0) for Netbox circuit ID 173 [production]
19:11 <eileen> civicrm upgraded from d13e6e0c to fc5c73db [production]
19:10 <ayounsi@cumin1001> START - Cookbook sre.network.debug for Netbox circuit ID 173 [production]
19:10 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host restbase1026.eqiad.wmnet [production]
19:09 <eileen> civicrm upgraded from d13e6e0c to fc5c73db [production]
19:07 <zabe@deploy1002> Finished scap: update interwiki cache (duration: 07m 08s) [production]
19:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1160 (T343718)', diff saved to https://phabricator.wikimedia.org/P51947 and previous config saved to /var/cache/conftool/dbconfig/20230829-190635-ladsgroup.json [production]
19:01 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudservices1006.eqiad.wmnet with reason: host reimage [production]
19:01 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host restbase1025.eqiad.wmnet [production]
19:00 <zabe@deploy1002> Started scap: update interwiki cache [production]
18:56 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudservices1006.eqiad.wmnet with reason: host reimage [production]
18:55 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host cloudservices1006.eqiad.wmnet with OS bullseye [production]
18:55 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudservices1006.eqiad.wmnet with OS bullseye [production]
18:54 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host cloudservices1006.eqiad.wmnet with OS bullseye [production]