301-350 of 10000 results (24ms)
2017-09-03 §
17:52 <elukey> depooled cp4024 (ulsfo upload) due to kernel errors in dmesg [production]
15:57 <ema> restart varnish backend on cp1073 (mailbox lag) [production]
09:41 <ema@neodymium> conftool action : set/pooled=yes; selector: name=cp4024.ulsfo.wmnet,service=varnish-fe [production]
09:41 <ema@neodymium> conftool action : set/pooled=yes; selector: name=cp4024.ulsfo.wmnet,service=nginx [production]
09:25 <ema> cp4024 is down, powercycled [production]
09:13 <_joe_> manually fixed value of /conftool/v1/pools/eqiad/api_appserver/apache2/mw1208.eqiad.wmnet in codfw to allow replication to restart [production]
00:59 <legoktm> legoktm@terbium:~$ mwscript extensions/WikimediaMaintenance/filebackend/setZoneAccess.php --wiki=hiwikiversity --backend=local-multiwrite # T174859 [production]
2017-09-01 §
16:18 <bblack> restart varnish backend on cp1074 (mailbox lag) [production]
15:15 <urandom> Restarting Cassandra: restbase-dev100[5-6]-{a,b} [production]
15:06 <urandom> Restarting Cassandra: restbase-dev1004-{a,b} [production]
15:00 <reedy@tin> Synchronized php-1.30.0-wmf.16/extensions/Popups/: T174724 (duration: 00m 46s) [production]
14:28 <marostegui> Add 150G to /srv partition on labsdb1001 [production]
13:42 <marostegui> Rename table pr_index on enwiki on db1089 - T174782 [production]
13:10 <marostegui> Stop MySQL on db1026 as it will be decommissioned - T174763 [production]
12:32 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Restore db1081 original weight - T168661 (duration: 00m 43s) [production]
12:19 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db1026 as it will be decommissioned - T174763 (duration: 00m 43s) [production]
12:17 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Remove db1026 as it will be decommissioned - T174763 (duration: 00m 43s) [production]
11:50 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase db1081 weight - T168661 (duration: 00m 43s) [production]
11:23 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1081 with low weight - T168661 (duration: 00m 44s) [production]
10:35 <elukey> stop puppet on thorium and disable root rsyncs - T174756 [production]
10:31 <marostegui> Upgrade MariaDB to 10.0.32 on db1081 - T168661 [production]
10:29 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1081 - T168661 (duration: 00m 43s) [production]
09:10 <godog> bounce graphite-web on graphite1001, problematic query? using all CPU [production]
09:04 <ema> lvs1007 upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882 [production]
08:59 <godog> depool restbase200[135] before reimage - T169939 [production]
07:22 <elukey> restart apache2 and hue on thorium, Analytics sites down, investigating [production]
07:06 <marostegui> Power reset db2044 as it is unresponsive - T174764 [production]
03:54 <krinkle@tin> Synchronized wmf-config/InitialiseSettings.php: I4dfc33f66c3 - Enable jQuery 3 on nlwiki sister projects (duration: 00m 43s) [production]
00:57 <tstarling@tin> Synchronized php-1.30.0-wmf.16/includes/parser/Parser.php: (no justification provided) (duration: 00m 44s) [production]
00:13 <RainbowSprinkles> gerrit: flushed all non-login caches, things might be sluggish for the next ~15mins or so [production]
2017-08-31 §
20:03 <demon@tin> rebuilt wikiversions.php and synchronized wikiversions files: (no justification provided) [production]
19:27 <mattflaschen@tin> Finished scap: Watchlist filters: Convert edit watchlist button to new UX and fix server-side tag filtering. T172030 (duration: 21m 23s) [production]
19:17 <cmjohnson1> powercycled mw1294 [production]
19:05 <mattflaschen@tin> Started scap: Watchlist filters: Convert edit watchlist button to new UX and fix server-side tag filtering. T172030 [production]
18:53 <mattflaschen@tin> Synchronized docroot/noc/conf/interwiki-labs.php.txt: Interwiki links on Beta Cluster: T69931 (duration: 00m 47s) [production]
18:51 <mattflaschen@tin> Synchronized wmf-config: Interwiki links on Beta Cluster: T69931 (duration: 00m 49s) [production]
17:50 <demon@tin> Synchronized php-1.30.0-wmf.16/skins/MinervaNeue/resources/skins.minerva.icons.images.scripts/watch.svg: sorta impromptu swat thing (duration: 00m 47s) [production]
16:35 <ejegg> re-enabled donation queue consumer and thank you mail sender [production]
15:25 <ejegg> updated civicrm from f00611d9aa3ae4a1ffbff70dcca90c7ae07b6c68 to 820bd98c7c23a851c38a75111c8c1532c781e198 [production]
15:20 <ejegg> updated SmashPig from 8f4388d407f313c1083cf96ed106efcc5d5a29f7 to 8eb98c10ff1129ea65dd81559656001c819cc12f [production]
15:19 <ejegg> turned off donation queue consumer and thank you job [production]
15:18 <elukey> restart zookeeper on conf200[2,3] for jvm security updates [production]
13:36 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Restore db1084 original weight - T168661 (duration: 00m 47s) [production]
13:10 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase db1084 weight - T168661 (duration: 00m 47s) [production]
13:07 <elukey> restart zookeeper on conf2001 for security updates (canary node) [production]
13:00 <moritzm> installing augeas security updates [production]
12:34 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1084 with low weight - T168661 (duration: 00m 47s) [production]
12:14 <marostegui> Upgrade MariaDB on db1084 - T168661 [production]
12:11 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1084 - T168661 (duration: 00m 47s) [production]
12:05 <akosiaris> reimage chlorine and kubernetes1004 T170119 [production]