3201-3250 of 10000 results (79ms)
2022-09-12 ยง
18:01 <inflatador> [WDQS Deploy] Tests passing following deploy of `wdqs1003` on canary `wdqs1003`; proceeding to rest of fleet [production]
17:57 <inflatador> [WDQS Deploy] Gearing up for deploy of wdqs `0.3.116`. Pre-deploy tests passing on canary `wdqs1003` [production]
17:43 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1127 (T314041)', diff saved to https://phabricator.wikimedia.org/P34532 and previous config saved to /var/cache/conftool/dbconfig/20220912-174301-ladsgroup.json [production]
17:42 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1127.eqiad.wmnet with reason: Maintenance [production]
17:42 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1127.eqiad.wmnet with reason: Maintenance [production]
17:42 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T314041)', diff saved to https://phabricator.wikimedia.org/P34531 and previous config saved to /var/cache/conftool/dbconfig/20220912-174239-ladsgroup.json [production]
17:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P34529 and previous config saved to /var/cache/conftool/dbconfig/20220912-172733-ladsgroup.json [production]
17:21 <bmansurov@deploy1002> Finished deploy [airflow-dags/research@b9be20d]: (no justification provided) (duration: 00m 09s) [production]
17:21 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b9be20d]: (no justification provided) [production]
17:12 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P34528 and previous config saved to /var/cache/conftool/dbconfig/20220912-171227-ladsgroup.json [production]
17:08 <cwhite> rebuilt raid on logstash2027 T316996 [production]
16:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T314041)', diff saved to https://phabricator.wikimedia.org/P34527 and previous config saved to /var/cache/conftool/dbconfig/20220912-165720-ladsgroup.json [production]
15:54 <bmansurov@deploy1002> Finished deploy [airflow-dags/research@b9be20d]: (no justification provided) (duration: 00m 09s) [production]
15:54 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b9be20d]: (no justification provided) [production]
15:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2118 (T314041)', diff saved to https://phabricator.wikimedia.org/P34524 and previous config saved to /var/cache/conftool/dbconfig/20220912-152920-ladsgroup.json [production]
15:29 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2118.codfw.wmnet with reason: Maintenance [production]
15:29 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2118.codfw.wmnet with reason: Maintenance [production]
15:28 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2108 (T314041)', diff saved to https://phabricator.wikimedia.org/P34523 and previous config saved to /var/cache/conftool/dbconfig/20220912-152858-ladsgroup.json [production]
15:18 <dancy@deploy1002> Installation of scap version "4.18.0" completed for 561 hosts [production]
15:17 <dancy@deploy1002> Installing scap version "4.18.0" for 561 hosts [production]
15:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P34522 and previous config saved to /var/cache/conftool/dbconfig/20220912-151352-ladsgroup.json [production]
14:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P34521 and previous config saved to /var/cache/conftool/dbconfig/20220912-145845-ladsgroup.json [production]
14:43 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2108 (T314041)', diff saved to https://phabricator.wikimedia.org/P34520 and previous config saved to /var/cache/conftool/dbconfig/20220912-144339-ladsgroup.json [production]
14:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2176 (T312863)', diff saved to https://phabricator.wikimedia.org/P34519 and previous config saved to /var/cache/conftool/dbconfig/20220912-141427-ladsgroup.json [production]
14:14 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2176.codfw.wmnet with reason: Maintenance [production]
14:14 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2176.codfw.wmnet with reason: Maintenance [production]
14:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2174 (T312863)', diff saved to https://phabricator.wikimedia.org/P34518 and previous config saved to /var/cache/conftool/dbconfig/20220912-141405-ladsgroup.json [production]
14:05 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1098.eqiad.wmnet [production]
14:02 <volans@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts wtp[1028-1030] [production]
14:02 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:01 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
13:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P34517 and previous config saved to /var/cache/conftool/dbconfig/20220912-135859-ladsgroup.json [production]
13:57 <btullis@cumin1001> START - Cookbook sre.hosts.reboot-single for host an-worker1098.eqiad.wmnet [production]
13:53 <volans@cumin1001> START - Cookbook sre.hosts.decommission for hosts wtp[1028-1030] [production]
13:50 <bmansurov@deploy1002> Finished deploy [airflow-dags/research@b9be20d]: (no justification provided) (duration: 00m 09s) [production]
13:49 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b9be20d]: (no justification provided) [production]
13:43 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P34516 and previous config saved to /var/cache/conftool/dbconfig/20220912-134353-ladsgroup.json [production]
13:43 <Lucas_WMDE> UTC afternoon backport+config window done [production]
13:40 <Lucas_WMDE> scap pull on mwdebug1001 to restore good code (confirmed that T317520 affects production) [production]
13:38 <marostegui@cumin1001> dbctl commit (dc=all): 'es1021 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34515 and previous config saved to /var/cache/conftool/dbconfig/20220912-133848-root.json [production]
13:35 <Lucas_WMDE> manually applying [[gerrit:830691]] on mwdebug1001 to test if T317520 affects production (expected to cause getExpensiveParserFunctionLimit-related logstash errors) [production]
13:33 <Lucas_WMDE> lucaswerkmeister-wmde@mwmaint1002:~$ printf 'https://en.wikipedia.org/static/images/%s\n' {mobile/copyright/wikipedia-ko-600k.svg,project-logos/kowiki-600k{,-1.5x,-2x}.png} | mwscript purgeList.php # T315127 [production]
13:30 <lucaswerkmeister-wmde@deploy1002> Synchronized static/images/mobile/copyright/: Config: [[gerrit:831212|Revert "kowiki: Add logo (legacy vector and vector-2022) for 600k articles" (T315127)]] (2/2; deleted file requires syncing whole directory) (duration: 03m 44s) [production]
13:28 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
13:28 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2174 (T312863)', diff saved to https://phabricator.wikimedia.org/P34514 and previous config saved to /var/cache/conftool/dbconfig/20220912-132846-ladsgroup.json [production]
13:26 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
13:26 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
13:26 <lucaswerkmeister-wmde@deploy1002> Synchronized static/images/project-logos/: Config: [[gerrit:831212|Revert "kowiki: Add logo (legacy vector and vector-2022) for 600k articles" (T315127)]] (1/2; deleted files require syncing whole directory) (duration: 03m 50s) [production]
13:25 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
13:23 <marostegui@cumin1001> dbctl commit (dc=all): 'es1021 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P34513 and previous config saved to /var/cache/conftool/dbconfig/20220912-132343-root.json [production]