151-200 of 10000 results (87ms)
2024-06-21 §
08:39 <aborrero@cumin1002> START - Cookbook sre.hosts.reboot-single for host cloudvirt1053.eqiad.wmnet [production]
08:14 <vgutierrez> restarting logrotate.service on cp[3068,3070-3071].esams.wmnet [production]
08:04 <akosiaris@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply [production]
08:04 <akosiaris@deploy1002> helmfile [eqiad] START helmfile.d/services/mobileapps: apply [production]
08:03 <akosiaris@deploy1002> helmfile [codfw] DONE helmfile.d/services/mobileapps: apply [production]
08:03 <akosiaris@deploy1002> helmfile [codfw] START helmfile.d/services/mobileapps: apply [production]
08:00 <akosiaris@deploy1002> helmfile [staging] DONE helmfile.d/services/mobileapps: apply [production]
08:00 <akosiaris@deploy1002> helmfile [staging] START helmfile.d/services/mobileapps: apply [production]
07:54 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1206 (re)pooling @ 100%: repool to fill up vslow/dump', diff saved to https://phabricator.wikimedia.org/P65297 and previous config saved to /var/cache/conftool/dbconfig/20240621-075404-arnaudb.json [production]
07:38 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1206 (re)pooling @ 75%: repool to fill up vslow/dump', diff saved to https://phabricator.wikimedia.org/P65296 and previous config saved to /var/cache/conftool/dbconfig/20240621-073858-arnaudb.json [production]
07:23 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1206 (re)pooling @ 50%: repool to fill up vslow/dump', diff saved to https://phabricator.wikimedia.org/P65295 and previous config saved to /var/cache/conftool/dbconfig/20240621-072353-arnaudb.json [production]
07:08 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1206 (re)pooling @ 25%: repool to fill up vslow/dump', diff saved to https://phabricator.wikimedia.org/P65294 and previous config saved to /var/cache/conftool/dbconfig/20240621-070847-arnaudb.json [production]
07:03 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1206 depool for debugging T368098', diff saved to https://phabricator.wikimedia.org/P65293 and previous config saved to /var/cache/conftool/dbconfig/20240621-070358-arnaudb.json [production]
04:51 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db2162 (T364069)', diff saved to https://phabricator.wikimedia.org/P65292 and previous config saved to /var/cache/conftool/dbconfig/20240621-045107-marostegui.json [production]
04:51 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2162.codfw.wmnet with reason: Maintenance [production]
04:50 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2162.codfw.wmnet with reason: Maintenance [production]
04:50 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2161 (T364069)', diff saved to https://phabricator.wikimedia.org/P65291 and previous config saved to /var/cache/conftool/dbconfig/20240621-045044-marostegui.json [production]
04:45 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2197.codfw.wmnet with reason: Maintenance [production]
04:45 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2197.codfw.wmnet with reason: Maintenance [production]
04:44 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2193 (T367856)', diff saved to https://phabricator.wikimedia.org/P65290 and previous config saved to /var/cache/conftool/dbconfig/20240621-044455-marostegui.json [production]
04:35 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P65289 and previous config saved to /var/cache/conftool/dbconfig/20240621-043537-marostegui.json [production]
04:29 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P65288 and previous config saved to /var/cache/conftool/dbconfig/20240621-042948-marostegui.json [production]
04:20 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P65287 and previous config saved to /var/cache/conftool/dbconfig/20240621-042030-marostegui.json [production]
04:14 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P65286 and previous config saved to /var/cache/conftool/dbconfig/20240621-041441-marostegui.json [production]
04:05 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2161 (T364069)', diff saved to https://phabricator.wikimedia.org/P65285 and previous config saved to /var/cache/conftool/dbconfig/20240621-040523-marostegui.json [production]
03:59 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2193 (T367856)', diff saved to https://phabricator.wikimedia.org/P65284 and previous config saved to /var/cache/conftool/dbconfig/20240621-035934-marostegui.json [production]
03:04 <ejegg> fundraising civicrm upgraded from 2e1db811 to 8a0b5bea [production]
01:45 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db2193 (T367856)', diff saved to https://phabricator.wikimedia.org/P65283 and previous config saved to /var/cache/conftool/dbconfig/20240621-014545-marostegui.json [production]
01:45 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2193.codfw.wmnet with reason: Maintenance [production]
01:45 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2193.codfw.wmnet with reason: Maintenance [production]
01:45 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2180 (T367856)', diff saved to https://phabricator.wikimedia.org/P65282 and previous config saved to /var/cache/conftool/dbconfig/20240621-014523-marostegui.json [production]
01:30 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P65281 and previous config saved to /var/cache/conftool/dbconfig/20240621-013016-marostegui.json [production]
01:15 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P65280 and previous config saved to /var/cache/conftool/dbconfig/20240621-011509-marostegui.json [production]
01:00 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2180 (T367856)', diff saved to https://phabricator.wikimedia.org/P65279 and previous config saved to /var/cache/conftool/dbconfig/20240621-010002-marostegui.json [production]
00:52 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1236 (T352010)', diff saved to https://phabricator.wikimedia.org/P65278 and previous config saved to /var/cache/conftool/dbconfig/20240621-005237-ladsgroup.json [production]
00:37 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P65277 and previous config saved to /var/cache/conftool/dbconfig/20240621-003730-ladsgroup.json [production]
00:22 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P65276 and previous config saved to /var/cache/conftool/dbconfig/20240621-002223-ladsgroup.json [production]
00:08 <mutante> [cp3072:~] $ sudo systemctl start varnishkafka-webrequest.service [production]
00:08 <mutante> [cp3067:~] $ sudo systemctl start logrotate [production]
00:07 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1236 (T352010)', diff saved to https://phabricator.wikimedia.org/P65275 and previous config saved to /var/cache/conftool/dbconfig/20240621-000716-ladsgroup.json [production]
00:00 <sukhe> restarting haproxy on cp3068 and cp3072 [production]
2024-06-20 §
23:47 <zabe@deploy1002> Finished scap: Update interwiki cache (duration: 10m 12s) [production]
23:36 <zabe@deploy1002> Started scap: Update interwiki cache [production]
23:34 <zabe> zabe@mwmaint1002:~$ mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=btmwiki --cluster=all 2>&1 | tee /tmp/btmwiki.UpdateSearchIndexConfig.log # T368038 [production]
23:33 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db2161 (T364069)', diff saved to https://phabricator.wikimedia.org/P65274 and previous config saved to /var/cache/conftool/dbconfig/20240620-233346-marostegui.json [production]
23:33 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2161.codfw.wmnet with reason: Maintenance [production]
23:33 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2161.codfw.wmnet with reason: Maintenance [production]
23:33 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2154 (T364069)', diff saved to https://phabricator.wikimedia.org/P65273 and previous config saved to /var/cache/conftool/dbconfig/20240620-233324-marostegui.json [production]
23:33 <zabe@deploy1002> Finished scap: Creating btmwiki (T368038) (duration: 12m 20s) [production]
23:20 <zabe@deploy1002> Started scap: Creating btmwiki (T368038) [production]