2701-2750 of 10000 results (37ms)
2015-12-23 §
19:22 <aaron@tin> Synchronized wmf-config/CommonSettings.php: Remove unused $wgMaxSquidPurgeTitles setting (duration: 00m 30s) [production]
18:38 <jynus> restart and reconfigure mysql at db2037 [production]
18:17 <mutante> bohrium - finish install, signing puppet certs [production]
17:11 <jynus> cloning s2 databases from dbstore2001 to dbstore2002 (s2 replication disabled on both) [production]
14:05 <jynus> restart and reconfigure mysql at db2064 [production]
13:26 <jynus> restart and reconfigure mysql at db2063 [production]
12:41 <jynus> reenabling event scheduler on db1046 (eventlogging m4-master) [production]
12:30 <jynus> restart and reconfigure mysql at db2056 [production]
12:14 <godog> upgrade cassandra on aqs1003 [production]
11:24 <jynus> reloading and reconfiguring mysql on db2049 [production]
11:11 <godog> roll-upgrade cassandra to 2.1.12 on aqs100[123] [production]
10:55 <jynus> rebooting and reconfiguring mysql on db2041 [production]
10:11 <jynus> restarting and reconfiguring mysql at db2035 [production]
09:52 <gwicke> rebuilding restbase1004 [production]
09:51 <gwicke> wiped & started boostrap on restbase1008 [production]
09:18 <gwicke> nodetool removenode e2813bb9-f1f2-4d21-ac19-95a7a35b4513 in preparation for adding 1004 to the cluster without bootstrap [production]
02:30 <l10nupdate@tin> ResourceLoader cache refresh completed at Wed Dec 23 02:30:25 UTC 2015 (duration 7m 1s) [production]
02:23 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.9) (duration: 09m 18s) [production]
00:40 <krenair@tin> Synchronized wmf-config/CommonSettings-labs.php: https://gerrit.wikimedia.org/r/260696 & https://gerrit.wikimedia.org/r/260699 (duration: 05m 28s) [production]
00:37 <mutante> mw1133 - powercycle [production]
00:36 <legoktm> manually fixed up stuck global rename of "RCJU-ArCJ" -> "Archives cantonales jurassiennes" [production]
00:31 <matt_flaschen> Ran UPDATE flow_workflow SET workflow_page_id = 41854369 WHERE workflow_wiki = 'enwiki' AND workflow_namespace = 5 AND workflow_title_text = 'Flow/Developer_test_page' AND workflow_page_id = 48099373; to work around DB inconsistency (T117812) [production]
2015-12-22 §
21:46 <gwicke> restbase1004: tune2fs -m 0 /dev/mapper/restbase1004--vg-srv [production]
21:45 <gwicke> restbase1004: restarted bootstrap [production]
21:22 <gwicke> restbase1003: restarting cassandra to clear up disk space from old stream [production]
21:11 <gwicke> restbase1008: restarting cassandra to clear up disk space from old stream [production]
18:36 <robh> silver returned to normal service, wikitech.w.o certificate renewed. [production]
18:26 <robh> silver puppet staying stalled during toollabs issue (we dont want to rehup silver web serivce) [production]
18:17 <robh> puppet disabled on silver, going to update wikitech.wikimedia.org certificate [production]
18:10 <jynus> disabling event scheduling on db1046 [production]
18:03 <jynus> rolling schema change (ALTER TABLE ENGINE=TokuDB) on m4-master (db1046) log (eventlogging) [production]
16:44 <godog> bounce cassandra on restbase1004, restart bootstrap [production]
16:42 <mutante> powercycling crashed mw1144 [production]
16:41 <jynus> converting dbstore2001 (delayed slave) into an actual delayed slave, adding redundancy to dbstore1002 [production]
16:40 <godog> bounce cassandra on restbase1003 [production]
16:15 <akosiaris> upgrade cassandra on maps-test2001 [production]
16:15 <akosiaris> upgrade cassandra on maps-test2002 [production]
15:53 <mutante> kafka1001,1002 - crit - eventlogging not running (?) [production]
15:52 <mutante> restbase1003 - disk space, restbase1008 - disk space, restbase1004 - cassandra cql refused [production]
15:23 <akosiaris> upgrade cassandra on maps-test2003 [production]
15:06 <jynus> restarting and reconfiguring mysql at dbstore2001 [production]
15:06 <mutante> labtestcontrol2001 - puppet had not been running for a while, a bunch of changes have been applied incl. keys and passwords [production]
15:04 <mutante> enabling puppet on labtestcontrol2001 [production]
15:04 <akosiaris> upgraded cassandra on maps-test2004 [production]
11:54 <apergos> salt packages with wmf packages precise running on ms-{bf}e* in esams; trusty running on analytics103* in eqiad; jessie running on restbase2* in codfw [production]
11:43 <godog> restart cassandra bootstrap on restbase1004 [production]
10:09 <jynus> online resizing /srv/postgres on labsdb1006 +100GB [production]
10:06 <hashar> Restarting Jenkins [production]
09:54 <apergos> precise and trusty salt packages with wmf patches deployed manually on dataset1001 and analytics1001, seem to work fine [production]
08:42 <jynus> restarting and reconfiguring mysql at db2036 [production]