501-550 of 10000 results (42ms)
2018-06-06 ยง
15:13 <_joe_> adding jobrunners, videoscalers to both pools with equal weight in codfw [production]
15:06 <oblivian@puppetmaster1001> conftool action : set/pooled=yes; selector: cluster=videoscaler,dc=codfw,service=nginx [production]
15:06 <urandom> upgrade Cassandra to 3.11.2, restbase1007-{b,c} - T178905 [production]
15:01 <awight@deploy1001> Started deploy [ores/deploy@65e979f]: ORES: new draft topic model; T176336 [production]
14:39 <oblivian@puppetmaster1001> conftool action : set/pooled=inactive; selector: cluster=videoscaler,dc=codfw,service=nginx,name=mw22(4[1-5]|5[3-8]|6[1-9]).* [production]
14:35 <oblivian@puppetmaster1001> conftool action : set/pooled=inactive; selector: cluster=videoscaler,dc=codfw,service=nginx,name=mw21(5[3-9]|6).* [production]
14:22 <andrewbogott> rebooting labvirt1009 [production]
14:21 <jynus> setting m2 on read write [production]
14:18 <jynus> setting gerrit on read only [production]
14:14 <jynus> starting s2-master switchover from db1051 to db1065 [production]
14:06 <andrewbogott> rebooting labvirt1003 [production]
13:58 <jynus> disabling puppet on db1051, db1065 [production]
13:35 <marostegui> Deploy schema change on db1096:3316 - T191316 T192926 T195193 T89737 [production]
13:35 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1096:3316 for alter table (duration: 00m 57s) [production]
13:06 <moritzm> installing elfutils security updates [production]
13:01 <akosiaris> starting slow rolling restart of all VMs on ganeti01.svc.codfw.wmnet [production]
12:59 <akosiaris> add +spec_ctrl to ganeti01.svc.codfw.wmnet cluster default cpu_type [production]
12:50 <jynus@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1084 with low load (duration: 00m 56s) [production]
12:24 <mobrovac@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Switch CirrusSearch jobs to EventBus for all wikis, file 2/2 - T189137 (duration: 00m 56s) [production]
12:23 <ppchelko@deploy1001> Finished deploy [cpjobqueue/deploy@c8d62da]: Enable cirrus for everything T190327 (duration: 00m 47s) [production]
12:23 <mobrovac@deploy1001> Synchronized wmf-config/jobqueue.php: Switch CirrusSearch jobs to EventBus for all wikis - T189137 (duration: 00m 57s) [production]
12:22 <ppchelko@deploy1001> Started deploy [cpjobqueue/deploy@c8d62da]: Enable cirrus for everything T190327 [production]
11:38 <jynus@deploy1001> Synchronized wmf-config/db-eqiad.php: Increase s4 weight for db1097 and db1103 (duration: 00m 56s) [production]
11:35 <jynus> stop and reimage db1084 [production]
11:27 <jynus@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1084 (duration: 00m 58s) [production]
10:19 <awight@deploy1001> Finished deploy [ores/deploy@65e979f]: ORES canary deployment to ores2002.codfw.wmnet; T176336 (duration: 03m 44s) [production]
10:16 <marostegui> Deploy schema change on dbstore1002:s6 - T191316 T192926 T195193 T89737 [production]
10:15 <awight@deploy1001> Started deploy [ores/deploy@65e979f]: ORES canary deployment to ores2002.codfw.wmnet; T176336 [production]
10:15 <awight@deploy1001> Finished deploy [ores/deploy@bf182e2]: ORES canary deployment to ores2002.codfw.wmnet; T176336 (duration: 00m 06s) [production]
10:15 <awight@deploy1001> Started deploy [ores/deploy@bf182e2]: ORES canary deployment to ores2002.codfw.wmnet; T176336 [production]
08:29 <marostegui> Reload haproxy on dbproxy1010 to repool labsdb1011 - T190704 [production]
08:25 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool all sanitariums masters - T190704 (duration: 00m 56s) [production]
08:24 <addshore@deploy1001> Synchronized wmf-config/Wikibase.php: wikidatawiki dispatching: [[gerrit:435648|dispatchMaxTime 720 (4 dispatchers at once)]] (duration: 00m 56s) [production]
08:14 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool all sanitariums masters - T190704 (duration: 00m 57s) [production]
07:48 <marostegui> Stop replication on all sanitarium masters to move labsdb1011 - T190704 [production]
07:30 <marostegui> Stop MySQL on labsdb1011 to install intel-microcode and reboot [production]
06:32 <marostegui> Deploy schema change on s6 codfw master (db2039), this will generate lag on s6 codfw - T191316 T192926 T195193 T89737 [production]
06:14 <marostegui> Stop slave on db2095:3316 to rebuild archive_insert and archive_update triggers - T192926 [production]
06:13 <ppchelko@deploy1001> Finished deploy [restbase/deploy@baa70b7]: Public release of feed availability endpoint T196402, take 2 (duration: 07m 13s) [production]
06:06 <ppchelko@deploy1001> Started deploy [restbase/deploy@baa70b7]: Public release of feed availability endpoint T196402, take 2 [production]
06:05 <ppchelko@deploy1001> Finished deploy [restbase/deploy@baa70b7]: Public release of feed availability endpoint T196402 (duration: 11m 45s) [production]
06:04 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool all sanitariums masters - T190704 (duration: 01m 09s) [production]
05:53 <ppchelko@deploy1001> Started deploy [restbase/deploy@baa70b7]: Public release of feed availability endpoint T196402 [production]
05:46 <marostegui> Reload haproxy on dbproxy1010 to depool labsdb1011 - T190704 [production]
05:31 <marostegui> Reload haproxy on dbproxy1010 to repool labsdb1010 - T190704 [production]
05:25 <marostegui> Restart MySQL on labsdb1010 [production]
05:24 <marostegui> Reload haproxy on dbproxy1010 to depool labsdb1010 - T190704 [production]
05:17 <marostegui> Deploy schema change on db1070 s5 primary master - T191316 T192926 T195193 [production]
04:44 <kartik@deploy1001> Finished deploy [cxserver/deploy@8ce20ba]: Update cxserver to 391d7b6 (Fixing T196462) (duration: 03m 06s) [production]
04:41 <kartik@deploy1001> Started deploy [cxserver/deploy@8ce20ba]: Update cxserver to 391d7b6 (Fixing T196462) [production]