4151-4200 of 10000 results (33ms)
2020-06-24 §
05:34 <marostegui@cumin2001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
05:33 <marostegui@cumin2001> START - Cookbook sre.hosts.decommission [production]
05:14 <marostegui> Remove grants from dbproxy1008 - T231280 T255406 [production]
05:03 <marostegui> Remove revision triggers from db1125:·3316 [production]
05:02 <marostegui@cumin2001> dbctl commit (dc=all): 'Depool db1085 for MCR schema change', diff saved to https://phabricator.wikimedia.org/P11643 and previous config saved to /var/cache/conftool/dbconfig/20200624-050235-marostegui.json [production]
04:53 <marostegui> Reload haproxy on dbproxy1012 and dbproxy1014 [production]
00:35 <ejegg> restarted fundraising jobs on main CiviCRM box [production]
00:33 <ejegg> updated Fundraising CiviCRM from f01b036128 to 52a32f2d66 [production]
2020-06-23 §
23:16 <wkandek> releases1002 is back after being moved to row D (T255590) [production]
23:11 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
22:35 <ejegg> disabled fundraising jobs on civi1001 for testing on civi2001 [production]
22:24 <wkandek@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
22:13 <AndyRussG> updated payments-wiki from 5fd4eb1519 to 28ad76dcd7 [production]
22:06 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
21:23 <wkandek@cumin1001> START - Cookbook sre.ganeti.makevm [production]
21:23 <dzahn@cumin1001> END (ERROR) - Cookbook sre.ganeti.makevm (exit_code=97) [production]
21:23 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
21:22 <wkandek@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
21:22 <wkandek@cumin1001> START - Cookbook sre.ganeti.makevm [production]
21:22 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
21:22 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
21:15 <wkandek@cumin1001> END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97) [production]
21:14 <wkandek@cumin1001> START - Cookbook sre.hosts.decommission [production]
20:31 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Migrate TemplateWizard from EventLogging to EventGate on all wikis - take 2 - T238230 (duration: 01m 06s) [production]
19:16 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Migrate TemplateWizard from EventLogging to EventGate on all wikis - T238230 (duration: 01m 05s) [production]
19:06 <brennen@deploy1001> rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.38 [production]
18:55 <mutante> gerrit1001 (prod) - restarting gerrit service to verify config changes [production]
18:53 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Migrate TemplateWizard from EventLogging to EventGate on group0 - T238230 (duration: 01m 06s) [production]
18:24 <reedy@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T254925 T246489 (duration: 01m 06s) [production]
18:04 <brennen@deploy1001> Finished scap: testwikis wikis to 1.35.0-wmf.38 (duration: 85m 53s) [production]
16:39 <brennen@deploy1001> Started scap: testwikis wikis to 1.35.0-wmf.38 [production]
16:01 <brennen> 1.35.0-wmf.38 was branched at a35f7318 for https://phabricator.wikimedia.org/T254175 [production]
15:47 <moritzm> prune nginx packages on mwdebug hosts T255565 [production]
15:37 <moritzm> prune nginx packages on mw1380-mw1412 T255565 [production]
15:28 <moritzm> installing libvpx security updates [production]
15:27 <mutante> removing ganeti VM xhgui1001 from eqiad row_A, will recreate in another row for rebalancing VMs between rows (T180761 T238098) [production]
15:26 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
15:18 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
15:12 <mutante> removing ganeti VM releases1002 in eqiad row_A - will recreate in another row to re-balance (T255590) [production]
15:12 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
15:10 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
14:56 <moritzm> failover ganeti master in eqiad to ganeti1011 [production]
14:55 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
14:50 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single [production]
14:48 <urbanecm@deploy1001> Synchronized private/PrivateSettings.php: T250887 (duration: 00m 58s) [production]
14:08 <mholloway-shell@deploy1001> Finished deploy [recommendation-api/deploy@db7fd80]: Update recommendation-api to 7e00177 (duration: 03m 13s) [production]
14:05 <mholloway-shell@deploy1001> Started deploy [recommendation-api/deploy@db7fd80]: Update recommendation-api to 7e00177 [production]
13:54 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:54 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
13:54 <jmm@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]