1201-1250 of 10000 results (23ms)
2014-06-26 §
04:45 <ori> mw1014 is sad, has filesystem issues: "Attempt to read block from filesystem resulted in short read while trying to open /tmp". Puppet can't run. Should be depooled. [production]
03:34 <LocalisationUpdate> ResourceLoader cache refresh completed at Thu Jun 26 03:33:19 UTC 2014 (duration 33m 18s) [production]
03:02 <LocalisationUpdate> completed (1.24wmf10) at 2014-06-26 03:01:43+00:00 [production]
02:33 <LocalisationUpdate> completed (1.24wmf9) at 2014-06-26 02:31:50+00:00 [production]
2014-06-25 §
23:43 <awight> updated crm from f3389daa94e9ad924175bdf0d5bc09c4a26aeb8c to e2fe03a9cd51e30206d9a1114d62dfbd6960816b [production]
23:27 <catrope> Finished scap: Updating Wikidata and TimedMediaHandler (duration: 04m 24s) [production]
23:23 <catrope> Started scap: Updating Wikidata and TimedMediaHandler [production]
21:22 <hashar> puppet fixed on gallium / lanthanum . It was missing a group definition. All fixed! Thanks Chase. [production]
20:53 <hashar> puppet broken on gallium.wikimedia.org and lanthanum.eqiad.wmnet . That is being looked at. [production]
20:34 <subbu> deployed parsoid 4ef9d6be [production]
19:38 <manybubbles> restarted Cirrus scripts after incident - the index rebuilds had to be completely restarted - sanity checking was simply paused [production]
18:54 <yurik> Synchronized wmf-config/PrivateSettings.php: Removed obsolete ZRMA user/pswd (duration: 01m 06s) [production]
18:46 <yurik> Finished scap: Removing ZeroRatedMobileAccess ext settings, depl latest JsonConfig/ZeroBanner/Portal (duration: 29m 09s) [production]
18:17 <yurik> Started scap: Removing ZeroRatedMobileAccess ext settings, depl latest JsonConfig/ZeroBanner/Portal [production]
17:41 <demon> Synchronized wmf-config/: Cirrus back on for wikis that had it before. Back to square 1 (duration: 00m 04s) [production]
17:29 <mwalker> updating fundraising tools from 5f3a7316b636c0723ce3fa353186d4041b662872 to cdc4b73bd59d27c8d386b6df629b1c574cfed85f [production]
17:06 <manybubbles> success! [production]
17:06 <manybubbles> Synchronized wmf-config/: try to fix cirrus (duration: 00m 04s) [production]
16:51 <andrewbogott> restarted apache on palladium -- _that_ helped [production]
16:49 <andrewbogott> it didn't help [production]
16:49 <andrewbogott> restarting puppetmaster on palladium [production]
16:42 <demon> Synchronized wmf-config/InitialiseSettings.php: Disable Cirrus everywhere but testwiki (duration: 00m 04s) [production]
16:23 <demon> Synchronized wmf-config/InitialiseSettings.php: Roll back previous Cirrus deploy (duration: 00m 05s) [production]
16:23 <demon> Synchronized wmf-config/CommonSettings.php: Roll back previous Cirrus deploy (duration: 00m 04s) [production]
16:16 <demon> Synchronized wmf-config/CommonSettings.php: I4c54357a: most remaining wikis getting Cirrus as primary (duration: 00m 04s) [production]
16:16 <demon> Synchronized wmf-config/InitialiseSettings.php: I4c54357a: most remaining wikis getting Cirrus as primary (duration: 00m 04s) [production]
15:27 <manybubbles> Synchronized php-1.24wmf9/extensions/Wikidata/: SWAT - fix rtl issue (duration: 00m 10s) [production]
15:23 <ottomata> reinstalling elastic1017,1018,1019 [production]
15:20 <manybubbles> Synchronized php-1.24wmf10/extensions/Wikidata/: SWAT - fix rtl issue (duration: 00m 12s) [production]
14:10 <Krinkle> Upgrade npm from v1.4.5 to v1.4.16 on integration-slave1001 and integration-slave1002 [production]
14:10 <Krinkle> Upgraded npm from v1.4.13 to v1.4.16 on integration-slave1003 to fix https://github.com/npm/npm/issues/5472 and repooling [production]
13:30 <Krinkle> Depooling integration-slave1003 as almost every other -npm build on this node fails due to corrupted ~/.npm cache [production]
12:52 <manybubbles> cirrus rebuild update: starting from mediawiki reindex step for all alphabetical wikis that have finished so far [production]
12:25 <hashar> Upgraded Zuul 9839edb..b7fc126 Brings patchset 20 of Zuul cloner ( https://review.openstack.org/#/c/70373/ ) [production]
12:02 <akosiaris> upgraded etherpad.wikimedia.org to etherpad-lite 1.4.0 [production]
11:12 <paravoid> switching inbound email for wikimedia.org to polonium/mchenry [production]
10:35 <_joe_> restarted lucene on search1016 as it was stuck there as well, once search1015 is up and running [production]
10:06 <_joe_> restarted lucene on search1015, it was stuck [production]
07:37 <springle> Synchronized wmf-config/db-eqiad.php: incremental LB bump on db1009 and db1021 traffic samplers (duration: 00m 07s) [production]
06:31 <springle> Synchronized wmf-config/db-eqiad.php: repool db1021 with traffic sampling (duration: 00m 09s) [production]
06:01 <springle> Synchronized wmf-config/db-eqiad.php: depool db1021, db1049 to normal load (duration: 00m 07s) [production]
05:05 <springle> Synchronized wmf-config/db-eqiad.php: repool db1049, warm up (duration: 00m 08s) [production]
02:50 <LocalisationUpdate> ResourceLoader cache refresh completed at Wed Jun 25 02:48:53 UTC 2014 (duration 48m 52s) [production]
02:39 <springle> xtrabackup clone db1005 to db1049 [production]
02:27 <LocalisationUpdate> completed (1.24wmf10) at 2014-06-25 02:25:57+00:00 [production]
02:19 <springle> Synchronized wmf-config/db-eqiad.php: depool db1049 (duration: 00m 11s) [production]
02:14 <LocalisationUpdate> completed (1.24wmf9) at 2014-06-25 02:13:27+00:00 [production]
00:57 <chasemp> added dns for wikimania 2015 (gerrit 140186) [production]
2014-06-24 §
23:28 <ori> apache-graceful-all was for Ifc9596cc7 [production]
23:28 <ori> gracefulled all apaches [production]