801-850 of 10000 results (31ms)
2016-04-19 §
02:06 <urandom> systemctl mask cassandra-b on restbase2004.codfw.wmnet (it should not be running) [production]
01:34 <mutante> restbase2004 - starting crashed cassandra-b service [production]
01:11 <mutante> restbase2004 - unit cassandra-b is failed [production]
00:42 <mutante> kraz - signing puppet certs, adding salt keys [production]
00:29 <mutante> kraz.codfw.wmnet - initial install, adding to site [production]
2016-04-18 §
23:33 <ori@tin> Synchronized wmf-config/CommonSettings.php: I1547834: Remove unused ['bits'] configuration (duration: 00m 27s) [production]
23:22 <YuviPanda> running sync-common on mw1145 to catch up on deploys [production]
23:17 <YuviPanda> rebooted mw1145 from mgmt [production]
23:14 <ori@tin> Synchronized wmf-config: I0ec3c015f: Update parser cache configuration for tag-based hashing (duration: 01m 48s) [production]
23:08 <jynus@tin> Synchronized wmf-config/db-codfw.php: Depool pc2006 (duration: 01m 01s) [production]
23:06 <ori@tin> Synchronized php-1.27.0-wmf.21/includes/objectcache/SqlBagOStuff.php: Ie9799f5ea: Allow tag names for SqlBagOStuff consistent hashing (duration: 04m 08s) [production]
22:52 <krinkle@tin> Synchronized docroot/noc/conf: noc: remote bits references (duration: 00m 32s) [production]
22:45 <jynus@tin> Synchronized wmf-config/db-codfw.php: Repool pc2005 (duration: 00m 33s) [production]
22:44 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Repool pc1005 (duration: 00m 26s) [production]
22:20 <chasemp> resetting labcontrol1001 puppet master with auth.conf which is fixing all the puppet clients in Labs [production]
21:33 <gehel> deploying latest wdqs [production]
21:29 <papaul> conf200[1-3] - signing puppet certs, salt-key, initial run [production]
20:58 <papaul> OS install on conf200[1-3] [production]
20:55 <mutante> furud.codfw - initial puppet install, signing certs [production]
18:43 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Depool pc1005 (duration: 00m 28s) [production]
18:42 <jynus@tin> Synchronized wmf-config/db-codfw.php: Depool pc2005 (duration: 00m 26s) [production]
18:37 <jynus@tin> Synchronized wmf-config/db-codfw.php: Repool pc2004 (duration: 00m 25s) [production]
18:27 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Repool pc1004 (duration: 00m 36s) [production]
17:31 <elukey> restarted kafkatee on oxygen after module upgrade (was stuck from this morning due to a kafka broker restart) [production]
14:21 <ottomata> upgraded pykafka on hafnium and restarted statsv [production]
14:04 <andrewbogott> puppet agent -tv on labvirt1002 which restarted nrpe and resolved icinga alerts [production]
14:02 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Depool pc1004 (duration: 00m 29s) [production]
14:01 <jynus@tin> Synchronized wmf-config/db-codfw.php: Depool pc2004 (duration: 00m 27s) [production]
11:30 <mobrovac> restbase restarting eqiad after https://gerrit.wikimedia.org/r/#/c/283958/ [production]
10:07 <_joe_> traffic from eqiad caches switched to codfw for restbase,citoid,cxserver [production]
09:43 <moritzm> installing openssh updates on jessie systems [production]
09:20 <_joe_> upgrading HHVM on terbium [production]
09:19 <godog> depool restbase100[56].eqiad.wmnet, about to get decomissioned [production]
09:17 <godog> repool restbase2006 after raid expansion [production]
09:08 <jynus@tin> Synchronized wmf-config/db-codfw.php: Update master comments in preparation for dc failover (duration: 00m 26s) [production]
09:07 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Update master comments in preparation for dc failover (duration: 01m 41s) [production]
07:02 <jynus> changing binlog_format to MIXED on db2018 [production]
2016-04-17 §
21:51 <urandom> Decommissioning restbase1005.eqiad.wmnet : T95253 [production]
21:50 <urandom> `systemctl mask cassandra' on restbase1006.eqiad.wmnet (node is decommissioned) : T95253 [production]
00:10 <urandom> Decommissioning restbase1006.eqiad.wmnet : T95253 [production]
2016-04-16 §
14:25 <andrewbogott> rebooting mw1132 — OOM [production]
14:24 <andrewbogott> rebooting mw1134 — OOM [production]
10:34 <jynus> recreating _counters table on s3 to solve replication issues [production]
10:08 <jynus> recreating hitcounter table on s3 to solve replication issues [production]
09:31 <volans> Skipped on dbstore1002 query from replica: 'DELETE FROM `bawiktionary`.`hitcounter`' [production]
09:31 <jynus> importing sys schema to db1075 [production]
06:44 <akosiaris> restart fermium to apply extra vcpus assignment [production]
06:23 <volans> Stopped and restarted replica on dbstore2002 for s3 to "unstuck" the replica [production]
02:31 <l10nupdate@tin> ResourceLoader cache refresh completed at Sat Apr 16 02:31:33 UTC 2016 (duration 9m 11s) [production]
02:22 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.21) (duration: 09m 03s) [production]