301-350 of 10000 results (53ms)
2018-10-08 §
10:41 <elukey> restart mcrouter on mw2201 with more verbose logging settings as test [production]
09:55 <moritzm> installing python3.5/python2.7 security updates [production]
09:51 <godog> rebuild sdc sdh sdj sdi on ms-be2041 with crc=1 finobt=0 - T199198 [production]
08:20 <marostegui> Disable gtid on es2 and es3 eqiad master [production]
08:20 <gehel@puppetmaster1001> conftool action : set/weight=15; selector: dc=codfw,cluster=wdqs,name=wdqs2001.codfw.wmnet [production]
08:20 <gehel@puppetmaster1001> conftool action : set/weight=15; selector: dc=codfw,cluster=wdqs,name=wdqs2002.codfw.wmnet [production]
07:50 <marostegui> Enabling replication eqiad -> codfw in preparation for DC failover [production]
07:40 <marostegui> Disable GTID on s1,s2,s3,s4,s6,s7,s8 eqiad masters in preparation for enabling replication eqiad -> codfw [production]
07:39 <_joe_> disabling puppet, doing etcd tests on lvs1006 [production]
07:38 <gehel@puppetmaster1001> conftool action : set/weight=15; selector: dc=codfw,cluster=wdqs,name=wdqs2002.eqiad.wmnet [production]
07:38 <gehel@puppetmaster1001> conftool action : set/weight=15; selector: dc=codfw,cluster=wdqs,name=wdqs2001.eqiad.wmnet [production]
07:38 <gehel> reducing relative weight of wdqs2003 in pybal - T206423 [production]
07:27 <banyek> enabling first time wmf-pt-kill on labsdb1010 [production]
07:20 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1092 with low weight - T205514 (duration: 01m 27s) [production]
07:00 <moritzm> installing git security updates [production]
2018-10-07 §
16:40 <dereckson> Reset user email for account "Dominic Mayers" (T206421) [production]
16:35 <elukey> run a script in tmux (my username) on mw2201 to poll the status of a mcrouter key/route every 10s using its admin api (very lightweight but kill if needed) [production]
14:52 <onimisionipe> repooling wdqs2003. Catched up on Lag and also Lag issues seems to be creeping on wdqs200[1|2] [production]
04:29 <SMalyshev> temp depooled wdqs2003 [production]
03:12 <ejegg> disabled all fundraising scheduled jobs - something that looks like disk issues on civi1001 [production]
2018-10-06 §
21:20 <gehel> repooling wdqs2003: catched up on updater lag [production]
20:43 <_joe_> restarting apache2 on puppetmaster1001 [production]
19:16 <onimisionipe> depooling wdqs2003 [production]
18:09 <elukey> restart Yarn Resource Manager on an-master1002 to force an-master1001 to take the active role back (failed over due to a zk conn issue) [production]
17:07 <onimisionipe> restarting wdqs-blazegraph on wdqs2003 [production]
13:48 <bblack> multatuli: update gdnsd package to 2.99.9930-beta-1+wmf1 [production]
13:47 <bblack> authdns1001: update gdnsd package to 2.99.9930-beta-1+wmf1 (correction to last msg) [production]
13:46 <bblack> authdns1001: update gdnsd package to 2.99.9161-beta-1+wmf1 [production]
12:57 <bblack> rebooting cp1076 [production]
12:49 <bblack> depool cp1076, apparently has disk issues [production]
2018-10-05 §
23:50 <bblack> <<<<<<< repooling eqiad edge caches, a few days ahead of intended switchback next Weds, to alleviate some traffic engineering concerns over the weekend >>>>>> [production]
20:48 <mutante> T191183 - it's still showing the error page as before but that isn't due to apache issues, it just needs additional ferm rules [production]
20:44 <mutante> gerrit - adding gerrit.wmfusercontent.org virtual host for avatars. applied first on gerrit2001, then on cobalt (T191183) [production]
20:03 <ejegg> updated fundraising CiviCRM from ebc2e0076c to 7a0d14015e [production]
19:48 <banyek> repooling labsdb1009 (T195747) [production]
19:44 <smalyshev@deploy1001> Finished deploy [wdqs/wdqs@f8776de]: Redeploy 1009 (duration: 00m 26s) [production]
19:44 <smalyshev@deploy1001> Started deploy [wdqs/wdqs@f8776de]: Redeploy 1009 [production]
18:37 <bblack> authdns2001: upgraded gdnsd to 2.99.9930-beta [production]
18:31 <bblack> gdnsd-2.99.9930-beta-1+wmf1 uploaded to stretch-wikimedia [production]
18:26 <mutante> icinga - noop on all servers, no change, puppet re-enabled, operations normal [production]
18:08 <mutante> disabling puppet on icinga for 5 min for extra safety before a change that should be noop [production]
17:58 <banyek> depooling labsdb1009 (T195747) [production]
17:50 <banyek> repooling labsdb1011 (T195747) [production]
17:12 <elukey> set etcd in codfw as read/write (was readonly) and eqiad as readonly (was read/write) [production]
14:57 <banyek> depooling labsdb1011 (T195747) [production]
14:56 <banyek> depooling labsdb1011 [production]
13:26 <banyek> adding wmf-pt-kill_2.2.20-1+wmf3 package for stretch [production]
13:25 <moritzm> installing python3.5/2.7 security updates [production]
13:02 <volans> upgraded spicerack to version 0.0.9 on sarin/neodymium/cumin* - T199079 [production]
12:13 <vgutierrez> Creating certcentral1001.eqiad.wmnet in ganeti - T206308 [production]