201-250 of 10000 results (27ms)
2015-12-26 §
18:39 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Emergency depool of db1022 (duration: 00m 30s) [production]
18:28 <jynus> disabling lag notifications for codfw (s6) [production]
18:01 <paravoid> cp3048: cleaned up /run/vmod_tbf/tbf.db/, kept a backup copy under ~faidon [production]
17:55 <paravoid> cp3048: service varnish-frontend stop (sending 429 to lots of people, T122453) [production]
05:38 <paravoid> rolling restart of hhvm jobrunners (T122069) [production]
02:31 <l10nupdate@tin> ResourceLoader cache refresh completed at Sat Dec 26 02:31:39 UTC 2015 (duration 6m 54s) [production]
02:24 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.9) (duration: 10m 39s) [production]
2015-12-25 §
15:50 <jynus> testing new mariadb packages on db2070 [production]
13:19 <jynus> setting db2018's binlog_format as MIXED [production]
10:51 <jynus> powercycle cp3010 [production]
09:17 <jynus> powercycling cp4007 (unresponsive to ssh, ping, serial console) [production]
02:29 <l10nupdate@tin> ResourceLoader cache refresh completed at Fri Dec 25 02:29:49 UTC 2015 (duration 6m 56s) [production]
02:22 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.9) (duration: 09m 47s) [production]
2015-12-24 §
23:28 <mutante> powercycled mw1114 [production]
23:27 <mutante> i just reset dra on mw1114 because it said it was in use and i didnt see a log yet :;p [production]
23:26 <robh> mw1114 spammed all icinga errors, system is outputting endless scroll of login prompt, not halting for input (like anohter session or crash cart is sending it, or an error) [production]
17:31 <gwicke> aqs: tweaked table properties for local_group_default_T_pageviews_per_article_flat: 2 months max DTCS window size, deflate compression [production]
17:20 <jynus> restarting and reconfiguring mysql at db2066 [production]
16:45 <jynus> restart and reconfigure mysql at db2059 [production]
16:11 <jynus> restart and mysql reconfguration of db2052 [production]
15:02 <jynus> restarting and reconfiguring mysql at db2045 [production]
14:11 <paravoid> powercycling mw1012, OOM'ed/stuck [production]
14:09 <paravoid> rolling restart of hhvm jobrunners (T122069) [production]
14:06 <jynus> restart and reconfigure mysql at db2038 [production]
12:32 <jynus> restart and reconfigure mysql at db2065 [production]
12:12 <jynus> restart and reconfiguring mysql for db2058 [production]
11:50 <jynus> restarting and reconfiguring mysql at db2051 [production]
11:28 <jynus> restarting and reconfiguring mysql at db2044 [production]
10:33 <jynus> restarting 's2' replication on dbstore200[12] after cloning [production]
02:30 <l10nupdate@tin> ResourceLoader cache refresh completed at Thu Dec 24 02:30:40 UTC 2015 (duration 6m 52s) [production]
02:23 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.9) (duration: 10m 03s) [production]
00:38 <gwicke> restbase1003: starting `nodetool cleanup` [production]
2015-12-23 §
23:31 <krenair@tin> Synchronized php-1.27.0-wmf.9/extensions/Graph/modules/ve-graph: https://gerrit.wikimedia.org/r/#/c/260868/ (duration: 00m 31s) [production]
19:59 <mutante> restbase1004 - puppet stopped and host key changed, what's up? [production]
19:42 <mutante> ran puppet on mw2112 [production]
19:41 <mutante> logstash1002 - started logstash service [production]
19:22 <aaron@tin> Synchronized wmf-config/CommonSettings.php: Remove unused $wgMaxSquidPurgeTitles setting (duration: 00m 30s) [production]
18:38 <jynus> restart and reconfigure mysql at db2037 [production]
18:17 <mutante> bohrium - finish install, signing puppet certs [production]
17:11 <jynus> cloning s2 databases from dbstore2001 to dbstore2002 (s2 replication disabled on both) [production]
14:05 <jynus> restart and reconfigure mysql at db2064 [production]
13:26 <jynus> restart and reconfigure mysql at db2063 [production]
12:41 <jynus> reenabling event scheduler on db1046 (eventlogging m4-master) [production]
12:30 <jynus> restart and reconfigure mysql at db2056 [production]
12:14 <godog> upgrade cassandra on aqs1003 [production]
11:24 <jynus> reloading and reconfiguring mysql on db2049 [production]
11:11 <godog> roll-upgrade cassandra to 2.1.12 on aqs100[123] [production]
10:55 <jynus> rebooting and reconfiguring mysql on db2041 [production]
10:11 <jynus> restarting and reconfiguring mysql at db2035 [production]
09:52 <gwicke> rebuilding restbase1004 [production]