|
2014-06-14
§
|
| 19:19 |
<paravoid> |
unmounting ms-be1001's sdj1, corrupted filesystem |
[production] |
| 18:47 |
<paravoid> |
rebooting ms-be1001, XFS: Internal error XFS_WANT_CORRUPTED_RETURN, lots of processes in D |
[production] |
| 03:08 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sat Jun 14 03:07:14 UTC 2014 (duration 7m 13s) |
[production] |
| 02:37 |
<LocalisationUpdate> |
completed (1.24wmf9) at 2014-06-14 02:36:35+00:00 |
[production] |
| 02:36 |
<bblack> |
enabled amssq43-46 frontends (esams text varnish) in pybal |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.24wmf8) at 2014-06-14 02:16:38+00:00 |
[production] |
| 00:46 |
<bblack> |
enabled amssq39-42 frontends (esams text varnish) in pybal |
[production] |
|
2014-06-13
§
|
| 22:01 |
<manybubbles> |
logstash1002 seems to be properly restoring nodes to itself. I'll monitor it for the next few minutes but I believe my work here is done. |
[production] |
| 21:55 |
<manybubbles> |
bouncing logstash1002 because it seems stuck. not sure why. no useful logs. |
[production] |
| 21:07 |
<bblack> |
turned on amssq35-38 text frontends in esams (in pybal) |
[production] |
| 20:57 |
<awight> |
update crm from c38296add61421f87e12cb5b4f3dd68bdf2340db to e52a4eb1bfab622f612dc84f687678fff1fdbc04 |
[production] |
| 20:23 |
<bblack> |
turned on amssq31-34 text frontends in esams |
[production] |
| 18:41 |
<mutante> |
DNS update - removing manutius' public IP |
[production] |
| 18:32 |
<mutante> |
shutting down manutius, decom |
[production] |
| 18:22 |
<ori> |
Synchronized php-1.24wmf9/extensions/Math: I498053de4: Fix the VisualEditor parts of Math-wmf9 with a working cherry pick of I7d5e1174 (duration: 00m 08s) |
[production] |
| 16:55 |
<hoo> |
Synchronized php-1.24wmf8/extensions/Wikidata/: Update Wikidata to fix JavaScript issues (duration: 00m 09s) |
[production] |
| 16:45 |
<hoo> |
Synchronized php-1.24wmf9/extensions/Wikidata/: Update Wikidata to fix JavaScript issues (duration: 00m 10s) |
[production] |
| 16:32 |
<Reedy> |
Finished creating mathoid tables on all wikis |
[production] |
| 16:26 |
<Reedy> |
Creating mathoid tables on all wikis |
[production] |
| 16:11 |
<mutante> |
manutius - decom, delete salt key, puppet cert, stopped services... |
[production] |
| 15:17 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Jun 13 15:16:09 UTC 2014 (duration 53m 11s) |
[production] |
| 14:59 |
<reedy> |
Synchronized wmf-config/: Disable MW_MATH_SOURCE for now (duration: 00m 15s) |
[production] |
| 14:46 |
<LocalisationUpdate> |
completed (1.24wmf9) at 2014-06-13 14:45:40+00:00 |
[production] |
| 14:36 |
<LocalisationUpdate> |
completed (1.24wmf8) at 2014-06-13 14:35:41+00:00 |
[production] |
| 13:00 |
<bblack> |
moved ge-3/0/0 - 3/0/15 from public to private vlan on cs2-esams (amssq31-46) |
[production] |
| 10:02 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1049 (duration: 00m 12s) |
[production] |
| 09:56 |
<paravoid> |
deactivating eqiad<->HE, excessive packet loss/latency |
[production] |
| 09:33 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1071 (duration: 00m 07s) |
[production] |
| 08:10 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1070, depool db1071 (duration: 00m 12s) |
[production] |
| 08:06 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1070, depool db1071 (duration: 00m 06s) |
[production] |
| 07:48 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1066, depool db1070 (duration: 00m 07s) |
[production] |
| 07:19 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1065, depool db1066 (duration: 00m 13s) |
[production] |
| 06:51 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1062, depool db1065 (duration: 00m 09s) |
[production] |
| 06:09 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1062 (duration: 00m 12s) |
[production] |
| 05:58 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1051 (duration: 00m 14s) |
[production] |
| 03:54 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Jun 13 03:53:17 UTC 2014 (duration 53m 16s) |
[production] |
| 03:12 |
<LocalisationUpdate> |
completed (1.24wmf9) at 2014-06-13 03:11:28+00:00 |
[production] |
| 02:35 |
<LocalisationUpdate> |
completed (1.24wmf8) at 2014-06-13 02:34:41+00:00 |
[production] |
| 00:45 |
<ori> |
Synchronized php-1.24wmf8/extensions/Math: Reverting Extension:Math to 1bb3bfa3b5656 (duration: 00m 05s) |
[production] |
| 00:44 |
<ori> |
Synchronized php-1.24wmf9/extensions/Math: Reverting Extension:Math to 1bb3bfa3b5656 (duration: 00m 06s) |
[production] |
| 00:41 |
<ori> |
removed Physikerwelt and Frédéric Wang from extension-Math group in Gerrit pending further inquiry into recent changes |
[production] |
| 00:38 |
<ori> |
Finished scap: fix any lingering inconsistencies in the state of the app servers (see https://gerrit.wikimedia.org/r/139089) (duration: 26m 59s) |
[production] |
| 00:11 |
<ori> |
Started scap: fix any lingering inconsistencies in the state of the app servers (see https://gerrit.wikimedia.org/r/139089) |
[production] |