9701-9750 of 10000 results (38ms)
2014-06-14 §
18:47 <paravoid> rebooting ms-be1001, XFS: Internal error XFS_WANT_CORRUPTED_RETURN, lots of processes in D [production]
03:08 <LocalisationUpdate> ResourceLoader cache refresh completed at Sat Jun 14 03:07:14 UTC 2014 (duration 7m 13s) [production]
02:37 <LocalisationUpdate> completed (1.24wmf9) at 2014-06-14 02:36:35+00:00 [production]
02:36 <bblack> enabled amssq43-46 frontends (esams text varnish) in pybal [production]
02:17 <LocalisationUpdate> completed (1.24wmf8) at 2014-06-14 02:16:38+00:00 [production]
00:46 <bblack> enabled amssq39-42 frontends (esams text varnish) in pybal [production]
2014-06-13 §
22:01 <manybubbles> logstash1002 seems to be properly restoring nodes to itself. I'll monitor it for the next few minutes but I believe my work here is done. [production]
21:55 <manybubbles> bouncing logstash1002 because it seems stuck. not sure why. no useful logs. [production]
21:07 <bblack> turned on amssq35-38 text frontends in esams (in pybal) [production]
20:57 <awight> update crm from c38296add61421f87e12cb5b4f3dd68bdf2340db to e52a4eb1bfab622f612dc84f687678fff1fdbc04 [production]
20:23 <bblack> turned on amssq31-34 text frontends in esams [production]
18:41 <mutante> DNS update - removing manutius' public IP [production]
18:32 <mutante> shutting down manutius, decom [production]
18:22 <ori> Synchronized php-1.24wmf9/extensions/Math: I498053de4: Fix the VisualEditor parts of Math-wmf9 with a working cherry pick of I7d5e1174 (duration: 00m 08s) [production]
16:55 <hoo> Synchronized php-1.24wmf8/extensions/Wikidata/: Update Wikidata to fix JavaScript issues (duration: 00m 09s) [production]
16:45 <hoo> Synchronized php-1.24wmf9/extensions/Wikidata/: Update Wikidata to fix JavaScript issues (duration: 00m 10s) [production]
16:32 <Reedy> Finished creating mathoid tables on all wikis [production]
16:26 <Reedy> Creating mathoid tables on all wikis [production]
16:11 <mutante> manutius - decom, delete salt key, puppet cert, stopped services... [production]
15:17 <LocalisationUpdate> ResourceLoader cache refresh completed at Fri Jun 13 15:16:09 UTC 2014 (duration 53m 11s) [production]
14:59 <reedy> Synchronized wmf-config/: Disable MW_MATH_SOURCE for now (duration: 00m 15s) [production]
14:46 <LocalisationUpdate> completed (1.24wmf9) at 2014-06-13 14:45:40+00:00 [production]
14:36 <LocalisationUpdate> completed (1.24wmf8) at 2014-06-13 14:35:41+00:00 [production]
13:00 <bblack> moved ge-3/0/0 - 3/0/15 from public to private vlan on cs2-esams (amssq31-46) [production]
10:02 <springle> Synchronized wmf-config/db-eqiad.php: depool db1049 (duration: 00m 12s) [production]
09:56 <paravoid> deactivating eqiad<->HE, excessive packet loss/latency [production]
09:33 <springle> Synchronized wmf-config/db-eqiad.php: repool db1071 (duration: 00m 07s) [production]
08:10 <springle> Synchronized wmf-config/db-eqiad.php: repool db1070, depool db1071 (duration: 00m 12s) [production]
08:06 <springle> Synchronized wmf-config/db-eqiad.php: repool db1070, depool db1071 (duration: 00m 06s) [production]
07:48 <springle> Synchronized wmf-config/db-eqiad.php: repool db1066, depool db1070 (duration: 00m 07s) [production]
07:19 <springle> Synchronized wmf-config/db-eqiad.php: repool db1065, depool db1066 (duration: 00m 13s) [production]
06:51 <springle> Synchronized wmf-config/db-eqiad.php: repool db1062, depool db1065 (duration: 00m 09s) [production]
06:09 <springle> Synchronized wmf-config/db-eqiad.php: depool db1062 (duration: 00m 12s) [production]
05:58 <springle> Synchronized wmf-config/db-eqiad.php: repool db1051 (duration: 00m 14s) [production]
03:54 <LocalisationUpdate> ResourceLoader cache refresh completed at Fri Jun 13 03:53:17 UTC 2014 (duration 53m 16s) [production]
03:12 <LocalisationUpdate> completed (1.24wmf9) at 2014-06-13 03:11:28+00:00 [production]
02:35 <LocalisationUpdate> completed (1.24wmf8) at 2014-06-13 02:34:41+00:00 [production]
00:45 <ori> Synchronized php-1.24wmf8/extensions/Math: Reverting Extension:Math to 1bb3bfa3b5656 (duration: 00m 05s) [production]
00:44 <ori> Synchronized php-1.24wmf9/extensions/Math: Reverting Extension:Math to 1bb3bfa3b5656 (duration: 00m 06s) [production]
00:41 <ori> removed Physikerwelt and Frédéric Wang from extension-Math group in Gerrit pending further inquiry into recent changes [production]
00:38 <ori> Finished scap: fix any lingering inconsistencies in the state of the app servers (see https://gerrit.wikimedia.org/r/139089) (duration: 26m 59s) [production]
00:11 <ori> Started scap: fix any lingering inconsistencies in the state of the app servers (see https://gerrit.wikimedia.org/r/139089) [production]
2014-06-12 §
23:35 <ori> Synchronized php-1.24wmf8/extensions/MobileFrontend: Re-syncing after submodule update (duration: 00m 06s) [production]
23:34 <ori> ran sync-common on mw1151 [production]
23:18 <catrope> Synchronized php-1.24wmf9/extensions/MobileFrontend: (no message) (duration: 00m 04s) [production]
23:17 <catrope> Synchronized php-1.24wmf8/extensions/MobileFrontend: (no message) (duration: 00m 05s) [production]
23:17 <catrope> Synchronized php-1.24wmf8/extensions/VisualEditor: (no message) (duration: 00m 04s) [production]
23:07 <Krinkle> integration-slave1003 is failing npm-test builds due to a cache corruption (filed as https://github.com/npm/npm/issues/5472). Manually cleared /mnt/home/jenkins-deploy/.npm/async on integration-slave1003.eqiad.wmflabs for now. [production]
23:05 <MaxSem> Purging PageImages data from Wikibooks and Wikisource [production]
22:59 <catrope> Synchronized wmf-config/: (no message) (duration: 00m 04s) [production]