1451-1500 of 10000 results (79ms)
2022-06-02 §
06:00 <Amir1> Starting s7 eqiad failover from db1181 to db1136 - T309617 [production]
05:55 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1146:3314 (T298560)', diff saved to https://phabricator.wikimedia.org/P29332 and previous config saved to /var/cache/conftool/dbconfig/20220602-055500-ladsgroup.json [production]
05:54 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1146.eqiad.wmnet with reason: Maintenance [production]
05:54 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1146.eqiad.wmnet with reason: Maintenance [production]
05:54 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298560)', diff saved to https://phabricator.wikimedia.org/P29331 and previous config saved to /var/cache/conftool/dbconfig/20220602-055452-ladsgroup.json [production]
05:39 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P29330 and previous config saved to /var/cache/conftool/dbconfig/20220602-053947-ladsgroup.json [production]
05:33 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1137 in x1 with minimal weight to test 10.6.8 T309679 ', diff saved to https://phabricator.wikimedia.org/P29329 and previous config saved to /var/cache/conftool/dbconfig/20220602-053340-marostegui.json [production]
05:24 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P29328 and previous config saved to /var/cache/conftool/dbconfig/20220602-052442-ladsgroup.json [production]
05:15 <ryankemper> T309720 Finished manual rolling restart of `cloudelastic` cluster to get new S3 plugin operational [production]
05:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2088 (s1 and s2) T309485', diff saved to https://phabricator.wikimedia.org/P29327 and previous config saved to /var/cache/conftool/dbconfig/20220602-051451-marostegui.json [production]
05:09 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298560)', diff saved to https://phabricator.wikimedia.org/P29326 and previous config saved to /var/cache/conftool/dbconfig/20220602-050937-ladsgroup.json [production]
05:05 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Set db1136 with weight 0 T309617', diff saved to https://phabricator.wikimedia.org/P29325 and previous config saved to /var/cache/conftool/dbconfig/20220602-050559-ladsgroup.json [production]
05:05 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s7 T309617 [production]
05:05 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 25 hosts with reason: Primary switchover s7 T309617 [production]
04:32 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1133.eqiad.wmnet with reason: Maintenance [production]
04:32 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1133.eqiad.wmnet with reason: Maintenance [production]
02:10 <krinkle@deploy1002> Synchronized docroot/noc/: Ic0e134c61d6 (duration: 03m 20s) [production]
02:04 <krinkle@deploy1002> Synchronized wmf-config/CommonSettings.php: Ic0e134c61d6 (duration: 03m 02s) [production]
01:49 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
01:48 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
01:48 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
01:47 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
01:38 <krinkle@deploy1002> Synchronized multiversion/: Id9b34b755230 no-op (duration: 03m 12s) [production]
01:37 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
01:36 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
01:36 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
01:35 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
01:15 <krinkle@deploy1002> Synchronized src/Profiler.php: I257b41a45 (duration: 03m 15s) [production]
01:15 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
01:14 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
01:14 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
01:13 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
01:09 <krinkle@deploy1002> Synchronized wmf-config/PhpAutoPrepend.php: Iebd29aaa (duration: 02m 57s) [production]
01:07 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
01:07 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
01:06 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
01:05 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
01:05 <krinkle@deploy1002> Synchronized src/Profiler.php: I93b3e43d32 (duration: 03m 16s) [production]
00:50 <krinkle@deploy1002> Synchronized wmf-config/MetaContactPages.php: Ief1368fd959f428 (duration: 02m 56s) [production]
00:46 <krinkle@deploy1002> Synchronized php-1.39.0-wmf.14/extensions/WikimediaMessages/: I5a700cd3648 (duration: 03m 01s) [production]
00:40 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
00:39 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
00:39 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
00:38 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
00:28 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
00:24 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
00:24 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
00:23 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
2022-06-01 §
22:13 <ryankemper> T309720 Downtimed cloudelastic until Monday while we perform maintenance across the next couple days (will manually lift downtime later) [production]
21:33 <bking@cumin1001> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: restart to enable S3 plugin - bking@cumin1001 - T309720 [production]