4051-4100 of 10000 results (70ms)
2020-03-03 ยง
15:56 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:55 <liw@deploy1001> Finished scap: group0 to 1.35.0-wmf.22 (duration: 24m 29s) [production]
15:49 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1096:3315 and db1096:3316 after reimage to buster T246604', diff saved to https://phabricator.wikimedia.org/P10596 and previous config saved to /var/cache/conftool/dbconfig/20200303-154913-marostegui.json [production]
15:47 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
15:45 <vgutierrez> Stopping pybal on lvs2009 to let lvs2010 get its traffic - T246686 [production]
15:45 <mutante> wtp1025 - scap pull as user cscott - testing sudo privs issue [production]
15:44 <vgutierrez> reimage lvs1014 with buster - T245984 [production]
15:43 <mutante> wtp1025 - scap pull [production]
15:35 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
15:34 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
15:31 <vgutierrez> Re-enable BGP in lvs1015 - T245984 [production]
15:31 <liw@deploy1001> Started scap: group0 to 1.35.0-wmf.22 [production]
15:30 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
15:22 <liw@deploy1001> Finished scap: testwiki to php-1.35.0-wmf.22 and rebuild l10n cache (duration: 71m 23s) [production]
15:20 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
15:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1096:3315 and db1096:3316 after reimage to buster T246604', diff saved to https://phabricator.wikimedia.org/P10595 and previous config saved to /var/cache/conftool/dbconfig/20200303-151805-marostegui.json [production]
15:15 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:15 <elukey@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-restart (exit_code=0) [production]
15:13 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Decrease a bit the weight for db1126', diff saved to https://phabricator.wikimedia.org/P10594 and previous config saved to /var/cache/conftool/dbconfig/20200303-150712-marostegui.json [production]
15:00 <vgutierrez> reimage lvs1015 with buster - T245984 [production]
14:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1096:3315 and db1096:3316 after reimage to buster T246604', diff saved to https://phabricator.wikimedia.org/P10591 and previous config saved to /var/cache/conftool/dbconfig/20200303-145230-marostegui.json [production]
14:44 <vgutierrez@cumin2001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
14:43 <vgutierrez@cumin2001> START - Cookbook sre.hosts.decommission [production]
14:43 <vgutierrez> running the decommission cookbook against lvs2001.codfw.wmnet - T246779 [production]
14:42 <vgutierrez> replace lvs2001 with lvs2007 - T196560 [production]
14:41 <addshore> START warm cache for db1111 & db1126 for Q20-25 million T219123 (pass 2) [production]
14:29 <vgutierrez> update puppet compiler facts [production]
14:28 <ottomata> postponing LVS for eventgate-analytics-external unti tomorrow [production]
14:13 <ottomata> beginning procedure to add LVS and discovery for eventgate-analytics-external - T233629 [production]
14:12 <elukey@cumin1001> START - Cookbook sre.elasticsearch.rolling-restart [production]
14:11 <liw@deploy1001> Started scap: testwiki to php-1.35.0-wmf.22 and rebuild l10n cache [production]
14:08 <liw@deploy1001> Pruned MediaWiki: 1.35.0-wmf.20 (duration: 15m 54s) [production]
14:08 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:06 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:55 <zpapierski@deploy1001> Finished deploy [wdqs/wdqs@8da3ae6]: (no justification provided) (duration: 10m 48s) [production]
13:54 <vgutierrez> reimage lvs1016 with buster - T245984 [production]
13:52 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:50 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:45 <zpapierski@deploy1001> Started deploy [wdqs/wdqs@8da3ae6]: (no justification provided) [production]
13:42 <zpapierski@deploy1001> Finished deploy [wdqs/wdqs@8da3ae6]: (no justification provided) (duration: 00m 26s) [production]
13:41 <zpapierski@deploy1001> Started deploy [wdqs/wdqs@8da3ae6]: (no justification provided) [production]
13:34 <vgutierrez> Re-enable BGP in lvs3005 - T245984 [production]
13:19 <akosiaris@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'changeprop' for release 'staging' . [production]
13:14 <akosiaris@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'changeprop' for release 'staging' . [production]
13:13 <akosiaris@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'changeprop' for release 'staging' . [production]
13:13 <akosiaris@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
13:12 <akosiaris@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
13:12 <akosiaris@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
13:12 <akosiaris@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'changeprop' for release 'staging' . [production]