1501-1550 of 10000 results (89ms)
2023-05-10 ยง
09:12 <jmm@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host cumin2002.codfw.wmnet [production]
09:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1137', diff saved to https://phabricator.wikimedia.org/P48101 and previous config saved to /var/cache/conftool/dbconfig/20230510-091001-ladsgroup.json [production]
09:09 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcontrol2001-dev.codfw.wmnet with reason: host reimage [production]
09:07 <marostegui@cumin1001> dbctl commit (dc=all): 'db2180 (re)pooling @ 25%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48100 and previous config saved to /var/cache/conftool/dbconfig/20230510-090755-root.json [production]
09:06 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcontrol2001-dev.codfw.wmnet with reason: host reimage [production]
09:01 <jmm@cumin1001> START - Cookbook sre.hosts.reboot-single for host cumin2002.codfw.wmnet [production]
09:01 <hashar> Gerrit restarted at version 3.5.6 | T336339 [production]
09:01 <marostegui@cumin1001> dbctl commit (dc=all): 'db2117 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P48099 and previous config saved to /var/cache/conftool/dbconfig/20230510-090119-root.json [production]
08:59 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1118 (T335845)', diff saved to https://phabricator.wikimedia.org/P48098 and previous config saved to /var/cache/conftool/dbconfig/20230510-085910-ladsgroup.json [production]
08:57 <hashar@deploy1002> Finished deploy [gerrit/gerrit@67ba7ab]: Gerrit to 3.5.6 on gerrit1001 | T336339 (duration: 00m 05s) [production]
08:57 <hashar@deploy1002> Started deploy [gerrit/gerrit@67ba7ab]: Gerrit to 3.5.6 on gerrit1001 | T336339 [production]
08:56 <hashar@deploy1002> Finished deploy [gerrit/gerrit@67ba7ab]: Gerrit to 3.5.6 on gerrit1001 | T336339 (duration: 00m 09s) [production]
08:56 <hashar@deploy1002> Started deploy [gerrit/gerrit@67ba7ab]: Gerrit to 3.5.6 on gerrit1001 | T336339 [production]
08:55 <hashar> Stopping Gerrit for 3.5.5 > 3.5.6 upgrade T336339 [production]
08:54 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1137', diff saved to https://phabricator.wikimedia.org/P48097 and previous config saved to /var/cache/conftool/dbconfig/20230510-085455-ladsgroup.json [production]
08:53 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1118 (T335845)', diff saved to https://phabricator.wikimedia.org/P48096 and previous config saved to /var/cache/conftool/dbconfig/20230510-085330-ladsgroup.json [production]
08:53 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1118.eqiad.wmnet with reason: Maintenance [production]
08:53 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1118.eqiad.wmnet with reason: Maintenance [production]
08:52 <marostegui@cumin1001> dbctl commit (dc=all): 'db2180 (re)pooling @ 10%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48095 and previous config saved to /var/cache/conftool/dbconfig/20230510-085250-root.json [production]
08:51 <aborrero@cumin1001> START - Cookbook sre.hosts.reimage for host cloudcontrol2001-dev.codfw.wmnet with OS bullseye [production]
08:49 <hashar@deploy1002> Finished deploy [gerrit/gerrit@67ba7ab]: Gerrit to 3.5.6 on gerrit2002 | T336339 (duration: 00m 07s) [production]
08:49 <hashar@deploy1002> Started deploy [gerrit/gerrit@67ba7ab]: Gerrit to 3.5.6 on gerrit2002 | T336339 [production]
08:48 <hashar> deploy1002: git reset `/srv/deployment/gerrit/gerrit` which had bunch of locally modified files for some reason # T336339 [production]
08:48 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1106.eqiad.wmnet with reason: Maintenance [production]
08:48 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1106.eqiad.wmnet with reason: Maintenance [production]
08:46 <marostegui@cumin1001> dbctl commit (dc=all): 'db2117 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P48094 and previous config saved to /var/cache/conftool/dbconfig/20230510-084614-root.json [production]
08:40 <volans@cumin1001> END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling update on A:netbox [production]
08:40 <volans@cumin1001> START - Cookbook sre.netbox.update-extras rolling update on A:netbox [production]
08:39 <volans@cumin1001> END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling update on A:netbox-canary [production]
08:39 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1137 (T335845)', diff saved to https://phabricator.wikimedia.org/P48093 and previous config saved to /var/cache/conftool/dbconfig/20230510-083948-ladsgroup.json [production]
08:39 <volans@cumin1001> START - Cookbook sre.netbox.update-extras rolling update on A:netbox-canary [production]
08:37 <marostegui@cumin1001> dbctl commit (dc=all): 'db2180 (re)pooling @ 5%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48092 and previous config saved to /var/cache/conftool/dbconfig/20230510-083745-root.json [production]
08:32 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1137 (T335845)', diff saved to https://phabricator.wikimedia.org/P48091 and previous config saved to /var/cache/conftool/dbconfig/20230510-083253-ladsgroup.json [production]
08:32 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1137.eqiad.wmnet with reason: Maintenance [production]
08:32 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1137.eqiad.wmnet with reason: Maintenance [production]
08:31 <marostegui@cumin1001> dbctl commit (dc=all): 'db2117 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P48090 and previous config saved to /var/cache/conftool/dbconfig/20230510-083109-root.json [production]
08:22 <marostegui@cumin1001> dbctl commit (dc=all): 'db2180 (re)pooling @ 3%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48089 and previous config saved to /var/cache/conftool/dbconfig/20230510-082240-root.json [production]
08:21 <hashar@deploy1002> Synchronized php: group1 wikis to 1.41.0-wmf.8 refs T330214 (duration: 05m 55s) [production]
08:16 <marostegui@cumin1001> dbctl commit (dc=all): 'db2117 (re)pooling @ 5%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P48088 and previous config saved to /var/cache/conftool/dbconfig/20230510-081605-root.json [production]
08:15 <hashar@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.41.0-wmf.8 refs T330214 [production]
08:14 <godog> re-enable eqsin remote syslog towards centrallog - T336345 [production]
08:07 <marostegui@cumin1001> dbctl commit (dc=all): 'db2180 (re)pooling @ 1%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48087 and previous config saved to /var/cache/conftool/dbconfig/20230510-080736-root.json [production]
08:03 <jmm@cumin2002> END (PASS) - Cookbook sre.netbox.restart-reboot (exit_code=0) rolling reboot on A:netbox [production]
08:01 <marostegui@cumin1001> dbctl commit (dc=all): 'db2117 (re)pooling @ 3%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P48086 and previous config saved to /var/cache/conftool/dbconfig/20230510-080100-root.json [production]
08:00 <marostegui@cumin1001> dbctl commit (dc=all): 'db1212 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48085 and previous config saved to /var/cache/conftool/dbconfig/20230510-080003-root.json [production]
07:59 <marostegui@cumin1001> dbctl commit (dc=all): 'db1112 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48084 and previous config saved to /var/cache/conftool/dbconfig/20230510-075957-root.json [production]
07:50 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) netbox.discovery.wmnet. on all recursors [production]
07:49 <jmm@cumin2002> START - Cookbook sre.dns.wipe-cache netbox.discovery.wmnet. on all recursors [production]
07:48 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.reimage (exit_code=99) for host testvm2005.codfw.wmnet with OS bookworm [production]
07:45 <marostegui@cumin1001> dbctl commit (dc=all): 'db2117 (re)pooling @ 1%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P48083 and previous config saved to /var/cache/conftool/dbconfig/20230510-074555-root.json [production]