1151-1200 of 10000 results (59ms)
2022-05-23 §
04:54 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1143.eqiad.wmnet with reason: Maintenance [production]
04:53 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 10:00:00 on db1143.eqiad.wmnet with reason: Maintenance [production]
04:40 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) [production]
04:40 <marostegui@cumin1001> Updating IPMI password on 1 hosts - marostegui@cumin1001 [production]
04:40 <marostegui@cumin1001> START - Cookbook sre.hosts.ipmi-password-reset [production]
04:40 <marostegui@cumin1001> END (FAIL) - Cookbook sre.hosts.ipmi-password-reset (exit_code=99) [production]
04:39 <marostegui@cumin1001> START - Cookbook sre.hosts.ipmi-password-reset [production]
2022-05-22 §
20:46 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:43 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:43 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:42 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:42 <krinkle@deploy1002> Synchronized wmf-config/: I14c5a9aa39 (duration: 00m 50s) [production]
20:41 <krinkle@deploy1002> Synchronized src/Profiler.php: I14c5a9aa39 (duration: 00m 49s) [production]
20:34 <krinkle@deploy1002> Synchronized lib/: I3882be35572 (duration: 00m 50s) [production]
20:32 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:32 <krinkle@deploy1002> Synchronized wmf-config/profiler.php: I3882be35572 (duration: 00m 51s) [production]
20:31 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:31 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:31 <krinkle@deploy1002> Synchronized src/XhguiSaverPdo.php: I3882be35572 (duration: 00m 50s) [production]
20:29 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
18:50 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T298560)', diff saved to https://phabricator.wikimedia.org/P28278 and previous config saved to /var/cache/conftool/dbconfig/20220522-185021-ladsgroup.json [production]
18:35 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P28277 and previous config saved to /var/cache/conftool/dbconfig/20220522-183516-ladsgroup.json [production]
18:20 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P28276 and previous config saved to /var/cache/conftool/dbconfig/20220522-182011-ladsgroup.json [production]
18:05 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T298560)', diff saved to https://phabricator.wikimedia.org/P28275 and previous config saved to /var/cache/conftool/dbconfig/20220522-180506-ladsgroup.json [production]
17:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298555)', diff saved to https://phabricator.wikimedia.org/P28274 and previous config saved to /var/cache/conftool/dbconfig/20220522-171444-ladsgroup.json [production]
14:49 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1138 (T298555)', diff saved to https://phabricator.wikimedia.org/P28273 and previous config saved to /var/cache/conftool/dbconfig/20220522-144855-ladsgroup.json [production]
14:48 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1138.eqiad.wmnet with reason: Maintenance [production]
14:48 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 10:00:00 on db1138.eqiad.wmnet with reason: Maintenance [production]
14:48 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298555)', diff saved to https://phabricator.wikimedia.org/P28272 and previous config saved to /var/cache/conftool/dbconfig/20220522-144847-ladsgroup.json [production]
14:27 <krinkle@deploy1002> Synchronized src/: Ia0a6d4794faaafc (duration: 00m 50s) [production]
14:23 <krinkle@deploy1002> Synchronized docroot/noc/: Ia0a6d4794faaafc (duration: 00m 50s) [production]
14:18 <krinkle@deploy1002> Synchronized wmf-config/: Ia0a6d4794faaafcb (2/2) (duration: 00m 42s) [production]
14:15 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
14:14 <krinkle@deploy1002> scap failed: average error rate on 3/8 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org for details) [production]
14:14 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
14:14 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
14:12 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
14:11 <krinkle@deploy1002> Synchronized multiversion/: Ia0a6d4794faaafcb (1/2) (duration: 00m 50s) [production]
14:07 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
14:03 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
14:03 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
14:02 <krinkle@deploy1002> Synchronized wmf-config/InitialiseSettings.php: I31b1bfb1808b9523 (duration: 00m 52s) [production]
13:59 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
13:44 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
13:40 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
13:40 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
13:36 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
13:28 <krinkle@deploy1002> Synchronized multiversion/: I3759179dba75a9419 (duration: 00m 53s) [production]
13:25 <krinkle@deploy1002> Synchronized wmf-config/CommonSettings.php: I97878f8e6 (duration: 00m 50s) [production]
13:21 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]