501-550 of 10000 results (27ms)
2015-12-28 §
21:26 <mutante> tin & mira: started salt minions that were in status stop/waiting [production]
21:25 <aaron@tin> Synchronized private/PrivateSettings.php: (no message) (duration: 00m 30s) [production]
21:20 <aaron@tin> Synchronized wmf-config/PrivateSettings.php: $wmfSwiftConfig convenience variable (duration: 00m 30s) [production]
20:59 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Repool db1050 & db1022 after emergency fix (duration: 00m 31s) [production]
20:52 <mutante> cygnus - starting salt-minion [production]
20:35 <yurik@tin> Synchronized php-1.27.0-wmf.9/extensions/Graph/modules/graph2.js: https://gerrit.wikimedia.org/r/#/c/261200/ (duration: 00m 31s) [production]
18:46 <jynus> importing wikishared from x1-master into analytics-slave and setting up replication [production]
18:28 <jynus> restarting and upgrading db1050, using the fact that it is depooled [production]
17:16 <paravoid> disabled varnish TBF and force-ran puppet on all cp* hosts (I12ea52165e125aaf4ed779399f34cff16d5cd140) [production]
16:38 <jynus> applying production-side replication filters for wikimania2017wiki on labs [production]
16:24 <krenair@tin> Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 30s) [production]
16:14 <krenair@tin> Synchronized dblists: (no message) (duration: 00m 29s) [production]
16:13 <krenair@tin> rebuilt wikiversions.php and synchronized wikiversions files: (no message) [production]
16:12 <krenair@tin> Synchronized w/static/images/project-logos/wikimania2017wiki.png: (no message) (duration: 00m 31s) [production]
16:12 <krenair@tin> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/260521/ (duration: 00m 30s) [production]
14:55 <jynus> cloning db1050's mysql data to db1022 [production]
14:15 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Emergency depool of db1050 (duration: 00m 31s) [production]
02:30 <l10nupdate@tin> ResourceLoader cache refresh completed at Mon Dec 28 02:30:47 UTC 2015 (duration 6m 58s) [production]
02:23 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.9) (duration: 09m 55s) [production]
2015-12-27 §
03:05 <YuviPanda> run nodetool clearsnapshot -- v3 and nodetool clearsnapshot -- v1 on maps-test2001 [production]
02:45 <YuviPanda> run drop keyspace v3; on csql on maps-test1001 for yurik [production]
02:42 <YuviPanda> run drop keyspace v1; on csql on maps-test1001 for yurik [production]
02:30 <l10nupdate@tin> ResourceLoader cache refresh completed at Sun Dec 27 02:30:29 UTC 2015 (duration 6m 59s) [production]
02:23 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.9) (duration: 09m 35s) [production]
2015-12-26 §
19:12 <paravoid> restarting varnish-frontend on cp3042 [production]
19:06 <jynus> setting db1030 as the new master of db2028 [production]
18:39 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Emergency depool of db1022 (duration: 00m 30s) [production]
18:28 <jynus> disabling lag notifications for codfw (s6) [production]
18:01 <paravoid> cp3048: cleaned up /run/vmod_tbf/tbf.db/, kept a backup copy under ~faidon [production]
17:55 <paravoid> cp3048: service varnish-frontend stop (sending 429 to lots of people, T122453) [production]
05:38 <paravoid> rolling restart of hhvm jobrunners (T122069) [production]
02:31 <l10nupdate@tin> ResourceLoader cache refresh completed at Sat Dec 26 02:31:39 UTC 2015 (duration 6m 54s) [production]
02:24 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.9) (duration: 10m 39s) [production]
2015-12-25 §
15:50 <jynus> testing new mariadb packages on db2070 [production]
13:19 <jynus> setting db2018's binlog_format as MIXED [production]
10:51 <jynus> powercycle cp3010 [production]
09:17 <jynus> powercycling cp4007 (unresponsive to ssh, ping, serial console) [production]
02:29 <l10nupdate@tin> ResourceLoader cache refresh completed at Fri Dec 25 02:29:49 UTC 2015 (duration 6m 56s) [production]
02:22 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.9) (duration: 09m 47s) [production]
2015-12-24 §
23:28 <mutante> powercycled mw1114 [production]
23:27 <mutante> i just reset dra on mw1114 because it said it was in use and i didnt see a log yet :;p [production]
23:26 <robh> mw1114 spammed all icinga errors, system is outputting endless scroll of login prompt, not halting for input (like anohter session or crash cart is sending it, or an error) [production]
17:31 <gwicke> aqs: tweaked table properties for local_group_default_T_pageviews_per_article_flat: 2 months max DTCS window size, deflate compression [production]
17:20 <jynus> restarting and reconfiguring mysql at db2066 [production]
16:45 <jynus> restart and reconfigure mysql at db2059 [production]
16:11 <jynus> restart and mysql reconfguration of db2052 [production]
15:02 <jynus> restarting and reconfiguring mysql at db2045 [production]
14:11 <paravoid> powercycling mw1012, OOM'ed/stuck [production]
14:09 <paravoid> rolling restart of hhvm jobrunners (T122069) [production]
14:06 <jynus> restart and reconfigure mysql at db2038 [production]