301-350 of 10000 results (41ms)
2017-01-03 §
10:14 <akosiaris> start enabling ntpd again across the fleet. Starting with cp boxes on ulsfo and esams [production]
09:23 <marostegui> stop MySQL dbstore2002 for maintenance - T151552 [production]
09:10 <marostegui> stop MySQL dbstore2001 for maintenance - T151552 [production]
08:21 <marostegui> Run optimize table on db1038 on all the revision,templatelinks and pagelinks tables - T154465 [production]
08:00 <marostegui> Run optimize table on a few large tables - db1015 - T153739 [production]
07:58 <elukey> chown www-data:www-data all the root:adm hhvm log files on mw codfw hosts (T132324) [production]
07:54 <marostegui> Run optimize table on a few large tables - db1044 - T153826 [production]
07:30 <marostegui> Stop mysql db2048 and db2034 for maintenance - https://phabricator.wikimedia.org/T149553 [production]
2017-01-02 §
23:09 <hoo> Removed 2fa from an account, per T154450 [production]
17:20 <ema> iridium: removed /var/log/account/pacct.2[0-9].gz to free up more disk space [production]
16:05 <ema> removing old kernels and kernel headers from iridium to free up some disk space [production]
13:24 <elukey> powercycled mw1280, not pingable and mgmt console frozen [production]
2017-01-01 §
02:23 <chasemp> labservices1001 'racadm serveraction hardreset' [production]
02:23 <godog> reboot labservices1001, unresponsive on console and MCE/temperature alerts found on lithium [production]
00:56 <filippo@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw1286.eqiad.wmnet,service=apache2 [production]
00:55 <bd808> Restarted logstash on logstash1001 (T154388) [production]
00:46 <filippo@puppetmaster1001> conftool action : set/pooled=no; selector: name=mw1286.eqiad.wmnet [production]
00:27 <godog> dump core file and restart varnish-frontend on cp2026 [production]
2016-12-31 §
14:32 <akosiaris> reenable puppet and ircecho on einsteinium. NTP disabling has propagated throughout the fleet [production]
14:08 <akosiaris> disable ircecho for a while to avoid NTP alert storm [production]
2016-12-29 §
23:19 <akosiaris> schedule downtime for ferm checks on kubernetes nodes. Some race between kubernetes + ferm, investigating [production]
21:19 <otto@tin> Finished deploy [eventstreams/deploy@4098bb4]: (no message) (duration: 01m 59s) [production]
21:17 <otto@tin> Starting deploy [eventstreams/deploy@4098bb4]: (no message) [production]
16:05 <_joe_> restarted HHVM on mw1279, stuck in HPHP::Treadmill::getAgeOldestRequest [production]
15:56 <akosiaris> merging https://gerrit.wikimedia.org/r/329597 for T154278 (IP throttle raise) [production]
15:55 <akosiaris@tin> Synchronized wmf-config/throttle.php: (no message) (duration: 00m 42s) [production]
12:20 <akosiaris> running sudo apt-get autoremove on labtestnet2001. Removing various older kernels [production]
12:19 <akosiaris> running sudo apt-get autoremove on labtestnet2001 [production]
05:22 <cwd> updated civicrm from 038e166269667e7b1c0e9ef54b06ae79b546c76e to f78c894ba6686f0b512e8120eb75e928ef8c92fe [production]
01:10 <cwd> updated payments-wiki from 5b4ced90352ec2c51186f93946840af24eaef290 to 21ea9bc4a46c96f84685b33ef06fd1b4c40e797e [production]
00:18 <cwd> updated smash-pig from c3eaadb737a02db27f3e97a8e60981c7182e877c to 3da597fdfc9718d10fae9c8e94825729a561d890 [production]
2016-12-28 §
23:13 <cwd> rolled back smashpig [production]
22:57 <cwd> updated smashpig from c3eaadb737a02db27f3e97a8e60981c7182e877c to 8597bd45fae3710bcb7c9bd0c592641723e52511 [production]
14:12 <mobrovac@tin> Finished deploy [trending-edits/deploy@c5d239b]: Add depooling and repooling to Trending Edits T144602 (duration: 01m 21s) [production]
14:11 <mobrovac@tin> Starting deploy [trending-edits/deploy@c5d239b]: Add depooling and repooling to Trending Edits T144602 [production]
14:07 <mobrovac@tin> Finished deploy [mobileapps/deploy@ae22656]: Add depooling and repooling to MCS T144602 (duration: 02m 34s) [production]
14:05 <mobrovac@tin> Starting deploy [mobileapps/deploy@ae22656]: Add depooling and repooling to MCS T144602 [production]
14:00 <mobrovac@tin> Finished deploy [electron-render/deploy@b2a820e]: Add depooling and repooling to PDFRender T144602 (duration: 01m 26s) [production]
13:59 <mobrovac@tin> Starting deploy [electron-render/deploy@b2a820e]: Add depooling and repooling to PDFRender T144602 [production]
13:51 <mobrovac@tin> Finished deploy [graphoid/deploy@151f26c]: Add depooling and repooling to Graphoid T144602 (duration: 01m 37s) [production]
13:49 <mobrovac@tin> Starting deploy [graphoid/deploy@151f26c]: Add depooling and repooling to Graphoid T144602 [production]
13:44 <mobrovac@tin> Finished deploy [cxserver/deploy@0279029]: Add depooling and repooling to CXServer T144602 (duration: 03m 22s) [production]
13:41 <mobrovac@tin> Starting deploy [cxserver/deploy@0279029]: Add depooling and repooling to CXServer T144602 [production]
13:41 <mobrovac@tin> Finished deploy [cxserver/deploy@0279029]: (no message) (duration: 00m 25s) [production]
13:40 <mobrovac@tin> Starting deploy [cxserver/deploy@0279029]: (no message) [production]
13:33 <mobrovac@tin> Finished deploy [citoid/deploy@da96f4b]: Add depooling and repooling to Citoid T144602 (duration: 01m 59s) [production]
13:31 <mobrovac@tin> Starting deploy [citoid/deploy@da96f4b]: Add depooling and repooling to Citoid T144602 [production]
13:21 <mobrovac@tin> Finished deploy [mathoid/deploy@79fdd56]: Add depooling and repooling to Mathoid T144602 (duration: 02m 31s) [production]
13:19 <mobrovac@tin> Starting deploy [mathoid/deploy@79fdd56]: Add depooling and repooling to Mathoid T144602 [production]
07:36 <legoktm> added reedy to "integration" gerrit group per T154207 [production]