2951-3000 of 10000 results (76ms)
2022-09-12 ยง
20:08 <samtar@deploy1002> Started scap: Backport for [[gerrit:831167|Mark spcomwiki and searchcomwiki as closed (T285685)]] [production]
20:07 <samtar@deploy1002> backport aborted: (duration: 03m 46s) [production]
20:06 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts theemin.codfw.wmnet [production]
20:06 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
20:04 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
19:59 <maryum> deployed security patch for T314245 [production]
19:59 <pt1979@cumin2002> START - Cookbook sre.hosts.decommission for hosts theemin.codfw.wmnet [production]
19:58 <mstyles@deploy1002> Synchronized php-1.39.0-wmf.28/extensions/PageTriage/includes/Api/ApiPageTriageAction.php: (no justification provided) (duration: 03m 42s) [production]
19:55 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2120 (T314041)', diff saved to https://phabricator.wikimedia.org/P34544 and previous config saved to /var/cache/conftool/dbconfig/20220912-195540-ladsgroup.json [production]
19:55 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2120.codfw.wmnet with reason: Maintenance [production]
19:55 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2120.codfw.wmnet with reason: Maintenance [production]
19:55 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2118 (T314041)', diff saved to https://phabricator.wikimedia.org/P34543 and previous config saved to /var/cache/conftool/dbconfig/20220912-195519-ladsgroup.json [production]
19:53 <sbassett> Deployed security patch for T311337 [production]
19:40 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2118', diff saved to https://phabricator.wikimedia.org/P34542 and previous config saved to /var/cache/conftool/dbconfig/20220912-194013-ladsgroup.json [production]
19:28 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1136 (T314041)', diff saved to https://phabricator.wikimedia.org/P34541 and previous config saved to /var/cache/conftool/dbconfig/20220912-192858-ladsgroup.json [production]
19:28 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1136.eqiad.wmnet with reason: Maintenance [production]
19:28 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1136.eqiad.wmnet with reason: Maintenance [production]
19:28 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1127 (T314041)', diff saved to https://phabricator.wikimedia.org/P34540 and previous config saved to /var/cache/conftool/dbconfig/20220912-192837-ladsgroup.json [production]
19:28 <bking@deploy1002> Finished deploy [wdqs/wdqs@e012d14]: 0.3.116 (duration: 02m 04s) [production]
19:26 <bking@deploy1002> Started deploy [wdqs/wdqs@e012d14]: 0.3.116 [production]
19:25 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2118', diff saved to https://phabricator.wikimedia.org/P34539 and previous config saved to /var/cache/conftool/dbconfig/20220912-192506-ladsgroup.json [production]
19:20 <dancy@deploy1002> Installation of scap version "4.19.0" completed for 561 hosts [production]
19:20 <dancy@deploy1002> Installing scap version "4.19.0" for 561 hosts [production]
19:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P34538 and previous config saved to /var/cache/conftool/dbconfig/20220912-191330-ladsgroup.json [production]
19:12 <dancy@deploy1002> Installation of scap version "4.18.0" completed for 561 hosts [production]
19:12 <dancy@deploy1002> Installing scap version "4.18.0" for 561 hosts [production]
19:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2118 (T314041)', diff saved to https://phabricator.wikimedia.org/P34537 and previous config saved to /var/cache/conftool/dbconfig/20220912-191000-ladsgroup.json [production]
19:08 <inflatador> [WDQS Deploy] Deploy complete. Successful test query placed on query.wikidata.org, there's no relevant criticals in Icinga, and Grafana looks good [production]
19:04 <ryankemper> [WCQS] Depooled `wcqs100[1,2]` while they catch up on ~1.5 days worth of lag (https://grafana.wikimedia.org/d/000000489/wikidata-query-service?orgId=1&var-cluster_name=wcqs&viewPanel=8&from=1662910789183&to=1663068616559) [production]
19:00 <inflatador> [WCQS Deploy] Test query passed on commons-query.wikimedia.org; WCQS deploy complete [production]
18:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P34536 and previous config saved to /var/cache/conftool/dbconfig/20220912-185823-ladsgroup.json [production]
18:56 <bking@deploy1002> Finished deploy [wdqs/wdqs@e012d14] (wcqs): Deploy 0.3.116 to WCQS (duration: 08m 01s) [production]
18:48 <bking@deploy1002> Started deploy [wdqs/wdqs@e012d14] (wcqs): Deploy 0.3.116 to WCQS [production]
18:43 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1127 (T314041)', diff saved to https://phabricator.wikimedia.org/P34535 and previous config saved to /var/cache/conftool/dbconfig/20220912-184317-ladsgroup.json [production]
18:37 <inflatador> [WDQS Deploy] Restarting `wdqs-categories` across lvs-managed hosts, one node at a time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 45 && systemctl restart wdqs-categories && sleep 45 && pool'` [production]
18:37 <inflatador> [WDQS Deploy] Restarted `wdqs-categories` across all test hosts simultaneously: `sudo -E cumin 'A:wdqs-test' 'systemctl restart wdqs-categories'` [production]
18:37 <inflatador> [WDQS Deploy] Restarted `wdqs-updater` across all hosts, 4 hosts at a time: `sudo -E cumin -b 4 'A:wdqs-all' 'systemctl restart wdqs-updater'` [production]
18:22 <bking@deploy1002> Finished deploy [wdqs/wdqs@e012d14]: 0.3.116 (duration: 07m 31s) [production]
18:14 <bking@deploy1002> Started deploy [wdqs/wdqs@e012d14]: 0.3.116 [production]
18:14 <dancy@deploy1002> Installation of scap version "4.16.0" completed for 561 hosts [production]
18:13 <dancy@deploy1002> Installing scap version "4.16.0" for 561 hosts [production]
18:08 <bking@deploy1002> Finished deploy [wdqs/wdqs@e012d14]: 0.3.116 (duration: 05m 37s) [production]
18:02 <bking@deploy1002> Started deploy [wdqs/wdqs@e012d14]: 0.3.116 [production]
18:01 <inflatador> [WDQS Deploy] Tests passing following deploy of `wdqs1003` on canary `wdqs1003`; proceeding to rest of fleet [production]
17:57 <inflatador> [WDQS Deploy] Gearing up for deploy of wdqs `0.3.116`. Pre-deploy tests passing on canary `wdqs1003` [production]
17:43 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1127 (T314041)', diff saved to https://phabricator.wikimedia.org/P34532 and previous config saved to /var/cache/conftool/dbconfig/20220912-174301-ladsgroup.json [production]
17:42 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1127.eqiad.wmnet with reason: Maintenance [production]
17:42 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1127.eqiad.wmnet with reason: Maintenance [production]
17:42 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T314041)', diff saved to https://phabricator.wikimedia.org/P34531 and previous config saved to /var/cache/conftool/dbconfig/20220912-174239-ladsgroup.json [production]
17:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P34529 and previous config saved to /var/cache/conftool/dbconfig/20220912-172733-ladsgroup.json [production]