5301-5350 of 10000 results (63ms)
2022-01-26 ยง
13:33 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1005.eqiad.wmnet [production]
13:28 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti1005.eqiad.wmnet [production]
13:26 <marostegui@cumin1001> dbctl commit (dc=all): 'db1137 (re)pooling @ 40%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19316 and previous config saved to /var/cache/conftool/dbconfig/20220126-132603-root.json [production]
13:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove recentchanges from s8 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P19315 and previous config saved to /var/cache/conftool/dbconfig/20220126-132600-marostegui.json [production]
13:21 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P19314 and previous config saved to /var/cache/conftool/dbconfig/20220126-132122-marostegui.json [production]
13:21 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T298559)', diff saved to https://phabricator.wikimedia.org/P19313 and previous config saved to /var/cache/conftool/dbconfig/20220126-132114-marostegui.json [production]
13:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1105:3312 (T298559)', diff saved to https://phabricator.wikimedia.org/P19311 and previous config saved to /var/cache/conftool/dbconfig/20220126-131959-marostegui.json [production]
13:19 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance [production]
13:19 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance [production]
13:19 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
13:19 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
13:19 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance [production]
13:19 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance [production]
13:19 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
13:19 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
13:17 <kharlan@deploy1002> helmfile [staging] DONE helmfile.d/services/linkrecommendation: sync on staging [production]
13:16 <kharlan@deploy1002> helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply on external [production]
13:16 <kharlan@deploy1002> helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply on internal [production]
13:16 <kharlan@deploy1002> helmfile [staging] START helmfile.d/services/linkrecommendation: apply on staging [production]
13:11 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1014.eqiad.wmnet [production]
13:10 <marostegui@cumin1001> dbctl commit (dc=all): 'db1137 (re)pooling @ 25%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19310 and previous config saved to /var/cache/conftool/dbconfig/20220126-131047-root.json [production]
13:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P19309 and previous config saved to /var/cache/conftool/dbconfig/20220126-130611-marostegui.json [production]
13:05 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti1014.eqiad.wmnet [production]
13:05 <marostegui@cumin1001> dbctl commit (dc=all): 'es1020 (re)pooling @ 100%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19308 and previous config saved to /var/cache/conftool/dbconfig/20220126-130527-root.json [production]
12:55 <marostegui@cumin1001> dbctl commit (dc=all): 'db1137 (re)pooling @ 20%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19307 and previous config saved to /var/cache/conftool/dbconfig/20220126-125543-root.json [production]
12:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T285149)', diff saved to https://phabricator.wikimedia.org/P19306 and previous config saved to /var/cache/conftool/dbconfig/20220126-125107-marostegui.json [production]
12:50 <marostegui@cumin1001> dbctl commit (dc=all): 'es1020 (re)pooling @ 75%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19305 and previous config saved to /var/cache/conftool/dbconfig/20220126-125023-root.json [production]
12:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1105:3311 (T285149)', diff saved to https://phabricator.wikimedia.org/P19304 and previous config saved to /var/cache/conftool/dbconfig/20220126-125001-marostegui.json [production]
12:50 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance [production]
12:49 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance [production]
12:49 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1119 (T285149)', diff saved to https://phabricator.wikimedia.org/P19303 and previous config saved to /var/cache/conftool/dbconfig/20220126-124953-marostegui.json [production]
12:46 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
12:45 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
12:45 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
12:44 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
12:43 <dcausse> UTC morning backport done [production]
12:41 <dcausse@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:757122|Correct wcqs event stream name]] (duration: 00m 51s) [production]
12:40 <marostegui@cumin1001> dbctl commit (dc=all): 'db1137 (re)pooling @ 10%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19302 and previous config saved to /var/cache/conftool/dbconfig/20220126-124040-root.json [production]
12:38 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es2024 (T300006)', diff saved to https://phabricator.wikimedia.org/P19301 and previous config saved to /var/cache/conftool/dbconfig/20220126-123839-ladsgroup.json [production]
12:35 <marostegui@cumin1001> dbctl commit (dc=all): 'es1020 (re)pooling @ 60%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P19300 and previous config saved to /var/cache/conftool/dbconfig/20220126-123520-root.json [production]
12:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P19299 and previous config saved to /var/cache/conftool/dbconfig/20220126-123448-marostegui.json [production]
12:34 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
12:33 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
12:32 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
12:32 <dcausse@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:757423|commonswiki: Add www.kew.org to the wgCopyUploadsDomains allowlist (T300101)]] (duration: 00m 51s) [production]
12:31 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
12:28 <kharlan@deploy1002> helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply on staging [production]
12:28 <kharlan@deploy1002> helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply on external [production]
12:28 <kharlan@deploy1002> helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply on internal [production]
12:28 <kharlan@deploy1002> helmfile [staging] START helmfile.d/services/linkrecommendation: apply on staging [production]