3851-3900 of 10000 results (75ms)
2022-09-14 §
01:18 <bking@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.UPGRADE (3 nodes at a time) for ElasticSearch cluster search_eqiad: elastic 6.8 -> 7.10 - bking@cumin1001 - T317686 [production]
01:16 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P34670 and previous config saved to /var/cache/conftool/dbconfig/20220914-011637-ladsgroup.json [production]
01:14 <ejegg> disabled delete_deleted_contacts job (will take effect when current job ends) [production]
01:01 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P34669 and previous config saved to /var/cache/conftool/dbconfig/20220914-010130-ladsgroup.json [production]
00:46 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1166 (T314041)', diff saved to https://phabricator.wikimedia.org/P34668 and previous config saved to /var/cache/conftool/dbconfig/20220914-004624-ladsgroup.json [production]
2022-09-13 §
23:46 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1166 (T314041)', diff saved to https://phabricator.wikimedia.org/P34667 and previous config saved to /var/cache/conftool/dbconfig/20220913-234607-ladsgroup.json [production]
23:46 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance [production]
23:45 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance [production]
23:45 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1157 (T314041)', diff saved to https://phabricator.wikimedia.org/P34666 and previous config saved to /var/cache/conftool/dbconfig/20220913-234546-ladsgroup.json [production]
23:30 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P34665 and previous config saved to /var/cache/conftool/dbconfig/20220913-233039-ladsgroup.json [production]
23:15 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P34664 and previous config saved to /var/cache/conftool/dbconfig/20220913-231533-ladsgroup.json [production]
23:13 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
23:13 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
23:12 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1202 (T314041)', diff saved to https://phabricator.wikimedia.org/P34663 and previous config saved to /var/cache/conftool/dbconfig/20220913-231257-ladsgroup.json [production]
23:03 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2182 (T314041)', diff saved to https://phabricator.wikimedia.org/P34662 and previous config saved to /var/cache/conftool/dbconfig/20220913-230317-ladsgroup.json [production]
23:03 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2182.codfw.wmnet with reason: Maintenance [production]
23:02 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2182.codfw.wmnet with reason: Maintenance [production]
23:02 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T314041)', diff saved to https://phabricator.wikimedia.org/P34661 and previous config saved to /var/cache/conftool/dbconfig/20220913-230255-ladsgroup.json [production]
23:00 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1157 (T314041)', diff saved to https://phabricator.wikimedia.org/P34660 and previous config saved to /var/cache/conftool/dbconfig/20220913-230026-ladsgroup.json [production]
22:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P34659 and previous config saved to /var/cache/conftool/dbconfig/20220913-225750-ladsgroup.json [production]
22:47 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P34658 and previous config saved to /var/cache/conftool/dbconfig/20220913-224749-ladsgroup.json [production]
22:42 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P34657 and previous config saved to /var/cache/conftool/dbconfig/20220913-224244-ladsgroup.json [production]
22:32 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P34656 and previous config saved to /var/cache/conftool/dbconfig/20220913-223241-ladsgroup.json [production]
22:30 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2149 (T314041)', diff saved to https://phabricator.wikimedia.org/P34655 and previous config saved to /var/cache/conftool/dbconfig/20220913-223025-ladsgroup.json [production]
22:30 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2149.codfw.wmnet with reason: Maintenance [production]
22:30 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2149.codfw.wmnet with reason: Maintenance [production]
22:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1202 (T314041)', diff saved to https://phabricator.wikimedia.org/P34654 and previous config saved to /var/cache/conftool/dbconfig/20220913-222738-ladsgroup.json [production]
22:19 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
22:19 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
22:19 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
22:19 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
22:17 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T314041)', diff saved to https://phabricator.wikimedia.org/P34653 and previous config saved to /var/cache/conftool/dbconfig/20220913-221734-ladsgroup.json [production]
22:16 <dancy> dancy@deploy1002$ rm /var/lib/deploy-mwdebug/pause [production]
22:15 <dancy@deploy1002> Sync cancelled. [production]
22:15 <dancy@deploy1002> dancy: testing synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
22:14 <dancy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
22:14 <dancy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
22:14 <dancy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
22:13 <dancy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
22:12 <dancy@deploy1002> Started scap: testing [production]
22:12 <dancy@deploy1002> Sync cancelled. [production]
22:11 <dancy@deploy1002> dancy: testing synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet [production]
22:11 <dancy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
22:11 <dancy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
22:10 <dancy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
22:08 <dancy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
22:07 <dancy@deploy1002> Started scap: testing [production]
22:07 <dancy@deploy1002> Sync cancelled. [production]
22:07 <dancy@deploy1002> dancy: testing synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
22:06 <dancy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]