4651-4700 of 10000 results (50ms)
2017-01-03 §
12:52 <akosiaris> reenabling ntpd on lvs boxes [production]
12:48 <akosiaris> reenabling ntpd on analytics boxes [production]
12:41 <moritzm> installing python security updates [production]
12:12 <moritzm> installing squid security updates [production]
12:07 <akosiaris> reenabling ntpd on pc eqiad & codfw boxes [production]
12:06 <akosiaris> reenabling ntpd on ganeti eqiad & codfw boxes [production]
11:54 <akosiaris> reenabling ntpd on wtp eqiad boxes [production]
11:52 <akosiaris> reenabling ntpd on logstash eqiad boxes [production]
11:51 <akosiaris> reenabling ntpd on db* eqiad boxes [production]
11:46 <akosiaris> reenabling ntpd on cobalt (gerrit) [production]
11:32 <moritzm> installing tar security updates on trusty hosts [production]
11:27 <gehel> upgrade liblogstash-gelf on deployment-elastic* - T150408 [production]
11:16 <gehel> upgrade lilogstash-gelf on relforge - T150408 [production]
11:13 <akosiaris> reenabling ntpd on db* codfw boxes [production]
11:07 <akosiaris> reenabling ntpd on wtp codfw boxes [production]
10:59 <akosiaris> reenabling ntpd on mw eqiad boxes [production]
10:53 <jynus> stopping mysql replication on db1035 (depooled) [production]
10:50 <akosiaris> reenabling ntpd on mw codfw boxes [production]
10:44 <akosiaris> reenabling ntpd on eqiad cp boxes [production]
10:39 <akosiaris> reenabling ntpd on codfw cp boxes [production]
10:14 <akosiaris> start enabling ntpd again across the fleet. Starting with cp boxes on ulsfo and esams [production]
09:23 <marostegui> stop MySQL dbstore2002 for maintenance - T151552 [production]
09:10 <marostegui> stop MySQL dbstore2001 for maintenance - T151552 [production]
08:21 <marostegui> Run optimize table on db1038 on all the revision,templatelinks and pagelinks tables - T154465 [production]
08:00 <marostegui> Run optimize table on a few large tables - db1015 - T153739 [production]
07:58 <elukey> chown www-data:www-data all the root:adm hhvm log files on mw codfw hosts (T132324) [production]
07:54 <marostegui> Run optimize table on a few large tables - db1044 - T153826 [production]
07:30 <marostegui> Stop mysql db2048 and db2034 for maintenance - https://phabricator.wikimedia.org/T149553 [production]
2017-01-02 §
23:09 <hoo> Removed 2fa from an account, per T154450 [production]
17:20 <ema> iridium: removed /var/log/account/pacct.2[0-9].gz to free up more disk space [production]
16:05 <ema> removing old kernels and kernel headers from iridium to free up some disk space [production]
13:24 <elukey> powercycled mw1280, not pingable and mgmt console frozen [production]
2017-01-01 §
02:23 <chasemp> labservices1001 'racadm serveraction hardreset' [production]
02:23 <godog> reboot labservices1001, unresponsive on console and MCE/temperature alerts found on lithium [production]
00:56 <filippo@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw1286.eqiad.wmnet,service=apache2 [production]
00:55 <bd808> Restarted logstash on logstash1001 (T154388) [production]
00:46 <filippo@puppetmaster1001> conftool action : set/pooled=no; selector: name=mw1286.eqiad.wmnet [production]
00:27 <godog> dump core file and restart varnish-frontend on cp2026 [production]
2016-12-31 §
14:32 <akosiaris> reenable puppet and ircecho on einsteinium. NTP disabling has propagated throughout the fleet [production]
14:08 <akosiaris> disable ircecho for a while to avoid NTP alert storm [production]
2016-12-29 §
23:19 <akosiaris> schedule downtime for ferm checks on kubernetes nodes. Some race between kubernetes + ferm, investigating [production]
21:19 <otto@tin> Finished deploy [eventstreams/deploy@4098bb4]: (no message) (duration: 01m 59s) [production]
21:17 <otto@tin> Starting deploy [eventstreams/deploy@4098bb4]: (no message) [production]
16:05 <_joe_> restarted HHVM on mw1279, stuck in HPHP::Treadmill::getAgeOldestRequest [production]
15:56 <akosiaris> merging https://gerrit.wikimedia.org/r/329597 for T154278 (IP throttle raise) [production]
15:55 <akosiaris@tin> Synchronized wmf-config/throttle.php: (no message) (duration: 00m 42s) [production]
12:20 <akosiaris> running sudo apt-get autoremove on labtestnet2001. Removing various older kernels [production]
12:19 <akosiaris> running sudo apt-get autoremove on labtestnet2001 [production]
05:22 <cwd> updated civicrm from 038e166269667e7b1c0e9ef54b06ae79b546c76e to f78c894ba6686f0b512e8120eb75e928ef8c92fe [production]
01:10 <cwd> updated payments-wiki from 5b4ced90352ec2c51186f93946840af24eaef290 to 21ea9bc4a46c96f84685b33ef06fd1b4c40e797e [production]