|
2014-06-16
§
|
| 10:23 |
<hoo> |
Touched all 1.24wmf8 extension/wikidata files and ran sync-common after that on mw1070 |
[production] |
| 10:18 |
<hoo> |
Synchronized php-1.24wmf8/extensions/Wikidata/: Update Wikidata to fix a suggester bug (duration: 00m 09s) |
[production] |
| 10:16 |
<godog> |
restarting swift-proxy-server on ms-fe3001 to test statsd metrics |
[production] |
| 10:13 |
<hoo> |
Synchronized php-1.24wmf9/extensions/Wikidata/: Update Wikidata to fix a suggester bug (duration: 00m 13s) |
[production] |
| 09:29 |
<apergos> |
restarted search1015 about 15 mns ago, it's now recovered afaict, restarted search1016, it's doing index setup now |
[production] |
| 03:00 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Jun 16 02:59:43 UTC 2014 (duration 59m 42s) |
[production] |
| 02:27 |
<LocalisationUpdate> |
completed (1.24wmf9) at 2014-06-16 02:26:05+00:00 |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.24wmf8) at 2014-06-16 02:14:38+00:00 |
[production] |
|
2014-06-15
§
|
| 17:44 |
<hoo> |
Synchronized php-1.24wmf8/extensions/Wikidata/: Touched various JavaScripts (duration: 00m 09s) |
[production] |
| 14:26 |
<Reedy> |
Job runners were restarted on tmh100[12] and are now processing jobs |
[production] |
| 14:15 |
<godog> |
extended palladium root partition by +20G |
[production] |
| 13:50 |
<_joe|away> |
restarted mw-job-runner on tmh1001 |
[production] |
| 10:02 |
<paravoid> |
nuked ms-be1001 sdj with zeros, reformatting and placing into production again |
[production] |
| 02:59 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sun Jun 15 02:58:21 UTC 2014 (duration 58m 20s) |
[production] |
| 02:27 |
<LocalisationUpdate> |
completed (1.24wmf9) at 2014-06-15 02:26:03+00:00 |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.24wmf8) at 2014-06-15 02:14:46+00:00 |
[production] |
|
2014-06-14
§
|
| 22:27 |
<bawolff> |
video scalers seem to have stopped doing webVideoTranscode jobs |
[production] |
| 20:24 |
<legoktm> |
ran "delete from ep_students where student_user_id =0 limit 1;" on enwiki for bug 66624 |
[production] |
| 20:10 |
<legoktm> |
ran "delete from ep_users_per_course where upc_user_id=0 limit 1" on enwiki for bug 66624 |
[production] |
| 19:19 |
<paravoid> |
unmounting ms-be1001's sdj1, corrupted filesystem |
[production] |
| 18:47 |
<paravoid> |
rebooting ms-be1001, XFS: Internal error XFS_WANT_CORRUPTED_RETURN, lots of processes in D |
[production] |
| 03:08 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sat Jun 14 03:07:14 UTC 2014 (duration 7m 13s) |
[production] |
| 02:37 |
<LocalisationUpdate> |
completed (1.24wmf9) at 2014-06-14 02:36:35+00:00 |
[production] |
| 02:36 |
<bblack> |
enabled amssq43-46 frontends (esams text varnish) in pybal |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.24wmf8) at 2014-06-14 02:16:38+00:00 |
[production] |
| 00:46 |
<bblack> |
enabled amssq39-42 frontends (esams text varnish) in pybal |
[production] |
|
2014-06-13
§
|
| 22:01 |
<manybubbles> |
logstash1002 seems to be properly restoring nodes to itself. I'll monitor it for the next few minutes but I believe my work here is done. |
[production] |
| 21:55 |
<manybubbles> |
bouncing logstash1002 because it seems stuck. not sure why. no useful logs. |
[production] |
| 21:07 |
<bblack> |
turned on amssq35-38 text frontends in esams (in pybal) |
[production] |
| 20:57 |
<awight> |
update crm from c38296add61421f87e12cb5b4f3dd68bdf2340db to e52a4eb1bfab622f612dc84f687678fff1fdbc04 |
[production] |
| 20:23 |
<bblack> |
turned on amssq31-34 text frontends in esams |
[production] |
| 18:41 |
<mutante> |
DNS update - removing manutius' public IP |
[production] |
| 18:32 |
<mutante> |
shutting down manutius, decom |
[production] |
| 18:22 |
<ori> |
Synchronized php-1.24wmf9/extensions/Math: I498053de4: Fix the VisualEditor parts of Math-wmf9 with a working cherry pick of I7d5e1174 (duration: 00m 08s) |
[production] |
| 16:55 |
<hoo> |
Synchronized php-1.24wmf8/extensions/Wikidata/: Update Wikidata to fix JavaScript issues (duration: 00m 09s) |
[production] |
| 16:45 |
<hoo> |
Synchronized php-1.24wmf9/extensions/Wikidata/: Update Wikidata to fix JavaScript issues (duration: 00m 10s) |
[production] |
| 16:32 |
<Reedy> |
Finished creating mathoid tables on all wikis |
[production] |
| 16:26 |
<Reedy> |
Creating mathoid tables on all wikis |
[production] |
| 16:11 |
<mutante> |
manutius - decom, delete salt key, puppet cert, stopped services... |
[production] |
| 15:17 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Jun 13 15:16:09 UTC 2014 (duration 53m 11s) |
[production] |
| 14:59 |
<reedy> |
Synchronized wmf-config/: Disable MW_MATH_SOURCE for now (duration: 00m 15s) |
[production] |
| 14:46 |
<LocalisationUpdate> |
completed (1.24wmf9) at 2014-06-13 14:45:40+00:00 |
[production] |
| 14:36 |
<LocalisationUpdate> |
completed (1.24wmf8) at 2014-06-13 14:35:41+00:00 |
[production] |
| 13:00 |
<bblack> |
moved ge-3/0/0 - 3/0/15 from public to private vlan on cs2-esams (amssq31-46) |
[production] |
| 10:02 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1049 (duration: 00m 12s) |
[production] |
| 09:56 |
<paravoid> |
deactivating eqiad<->HE, excessive packet loss/latency |
[production] |
| 09:33 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1071 (duration: 00m 07s) |
[production] |
| 08:10 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1070, depool db1071 (duration: 00m 12s) |
[production] |
| 08:06 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1070, depool db1071 (duration: 00m 06s) |
[production] |
| 07:48 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1066, depool db1070 (duration: 00m 07s) |
[production] |