2251-2300 of 10000 results (44ms)
2022-02-01 §
00:18 <ryankemper> [WDQS Deploy] Deploy complete. Successful test query placed on query.wikidata.org, there's no relevant criticals in Icinga, and Grafana looks good [production]
00:11 <catrope@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:758033|Lower The Wikipedia Library extension edit count (T288070)]] (duration: 00m 50s) [production]
00:11 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
00:10 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
00:10 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
00:09 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
2022-01-31 §
23:50 <dduvall@deploy1002> helmfile [eqiad] DONE helmfile.d/services/blubberoid: sync on production [production]
23:50 <dduvall@deploy1002> helmfile [eqiad] DONE helmfile.d/services/blubberoid: apply on staging [production]
23:50 <dduvall@deploy1002> helmfile [eqiad] START helmfile.d/services/blubberoid: apply on production [production]
23:49 <dduvall@deploy1002> helmfile [codfw] DONE helmfile.d/services/blubberoid: sync on production [production]
23:49 <dduvall@deploy1002> helmfile [codfw] DONE helmfile.d/services/blubberoid: apply on staging [production]
23:49 <dduvall@deploy1002> helmfile [codfw] START helmfile.d/services/blubberoid: apply on production [production]
23:44 <dduvall@deploy1002> helmfile [staging] DONE helmfile.d/services/blubberoid: sync on staging [production]
23:44 <dduvall@deploy1002> helmfile [staging] DONE helmfile.d/services/blubberoid: apply on production [production]
23:44 <dduvall@deploy1002> helmfile [staging] START helmfile.d/services/blubberoid: apply on staging [production]
23:31 <inflatador> [WCQS Deploy] Restarted `wcqs-updater` across all hosts: `sudo cumin -b 6 'wcqs*' 'sudo systemctl restart wcqs-updater'` [production]
23:29 <bking@deploy1002> Finished deploy [wdqs/wdqs@f0287fb] (wcqs): Deploy 0.3.101 to WCQS (duration: 02m 39s) [production]
23:28 <inflatador> [WCQS Deploy] Tests look good following deploy of `0.3.101` to canary `wcqs1002.eqiad.wmnet`, proceeding to rest of fleet [production]
23:26 <bking@deploy1002> Started deploy [wdqs/wdqs@f0287fb] (wcqs): Deploy 0.3.101 to WCQS [production]
23:17 <inflatador> [WDQS Deploy] Restarting `wdqs-categories` across lvs-managed hosts, one node at a time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 45 && systemctl restart wdqs-categories && sleep 45 && pool'` [production]
23:16 <inflatador> [WDQS Deploy] Restarted `wdqs-categories` across all test hosts simultaneously: `sudo -E cumin 'A:wdqs-test' 'systemctl restart wdqs-categories'` [production]
23:16 <inflatador> [WDQS Deploy] Restarted `wdqs-updater` across all hosts, 4 hosts at a time: `sudo -E cumin -b 4 'A:wdqs-all' 'systemctl restart wdqs-updater'` [production]
23:12 <bking@deploy1002> Finished deploy [wdqs/wdqs@f0287fb]: 0.3.101 (duration: 08m 18s) [production]
23:06 <inflatador> [WDQS Deploy] Tests passing following deploy of 0.3.101 on canary `wdqs1003`; proceeding to rest of fleet [production]
23:04 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudnet2004-dev.codfw.wmnet with OS bullseye [production]
23:03 <bking@deploy1002> Started deploy [wdqs/wdqs@f0287fb]: 0.3.101 [production]
22:43 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
22:42 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
22:42 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
22:40 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
22:38 <urbanecm> Deploy security patch for T298312 [production]
22:12 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudnet2004-dev.codfw.wmnet with OS bullseye [production]
21:15 <mutante> installed bullseye on new VM etherpad1003, signing puppet certs for etherpad1003.eqiad.wmnet - puppet error expected until we add the role (T300568) [production]
21:01 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
20:54 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
20:54 <ejegg> updated payments-wiki from a4b21e52 to 933e8669 [production]
20:50 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host etherpad1003.eqiad.wmnet [production]
20:39 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm for new host etherpad1003.eqiad.wmnet [production]
20:33 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-staging2002.mgmt.codfw.wmnet with reboot policy FORCED [production]
20:24 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host ml-staging2002.mgmt.codfw.wmnet with reboot policy FORCED [production]
20:21 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-staging2001.mgmt.codfw.wmnet with reboot policy FORCED [production]
20:14 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host ml-staging2001.mgmt.codfw.wmnet with reboot policy FORCED [production]
20:13 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-staging2001.mgmt.codfw.wmnet with reboot policy FORCED [production]
20:12 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host ml-staging2001.mgmt.codfw.wmnet with reboot policy FORCED [production]
20:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298559)', diff saved to https://phabricator.wikimedia.org/P19707 and previous config saved to /var/cache/conftool/dbconfig/20220131-201118-marostegui.json [production]
20:10 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-staging2001.mgmt.codfw.wmnet with reboot policy FORCED [production]
20:10 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host ml-staging2001.mgmt.codfw.wmnet with reboot policy FORCED [production]
20:09 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-staging2001.mgmt.codfw.wmnet with reboot policy FORCED [production]
20:09 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host ml-staging2001.mgmt.codfw.wmnet with reboot policy FORCED [production]
20:07 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-staging2001.mgmt.codfw.wmnet with reboot policy FORCED [production]