|
2014-06-18
§
|
| 08:26 |
<_joe_> |
disk is gone, powering down ms-be1007, opening ticket for disk replacement |
[production] |
| 08:24 |
<_joe_> |
stopped swift on ms-be1007, unmounting volume to check for repair |
[production] |
| 06:01 |
<springle> |
restarted gmetad on nickel while unbreaking the mysql graphs I broke on ganglia |
[production] |
| 04:30 |
<ori> |
enabled puppet on polonium (was disabled but nothing in SAL) |
[production] |
| 02:59 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Jun 18 02:58:22 UTC 2014 (duration 58m 21s) |
[production] |
| 02:26 |
<LocalisationUpdate> |
completed (1.24wmf9) at 2014-06-18 02:25:03+00:00 |
[production] |
| 02:23 |
<MaxSem> |
searchidx1001 outta sync - running sync-common |
[production] |
| 02:14 |
<LocalisationUpdate> |
completed (1.24wmf8) at 2014-06-18 02:13:34+00:00 |
[production] |
| 02:05 |
<Krinkle> |
Nevermind, graphite.wikimedia.org going down is due to overload which recovers eventually (it just has). Has become SNAFU/FIXME. |
[production] |
| 02:02 |
<Krinkle> |
graphite.wikimedia.org is down with HTTP 502 Bad Gateway errors |
[production] |
| 01:49 |
<ori> |
puppet freshness on tungsten and stat1001 can be fixed with https://gerrit.wikimedia.org/r/#/c/140269/ |
[production] |
|
2014-06-17
§
|
| 20:19 |
<maxsem> |
Synchronized php-1.24wmf9/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/#/c/140178/ (duration: 00m 04s) |
[production] |
| 20:17 |
<maxsem> |
Synchronized php-1.24wmf8/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/#/c/140178/ (duration: 00m 05s) |
[production] |
| 20:01 |
<hoo> |
Synchronized php-1.24wmf9/extensions/Wikidata/: Update Wikidata to fix editing site links (duration: 00m 24s) |
[production] |
| 18:23 |
<reedy> |
Synchronized docroot and w: (no message) (duration: 00m 16s) |
[production] |
| 18:22 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Non Wikipedias to 1.24wmf9 |
[production] |
| 18:05 |
<demon> |
Synchronized wmf-config/PoolCounterSettings-eqiad.php: Limit regex searches before they start landing on wikis (duration: 00m 04s) |
[production] |
| 16:32 |
<bblack> |
enabled amssq31-46 esams text frontend varnishes in pybal (were misconfigured; wrong domainname) |
[production] |
| 15:18 |
<manybubbles> |
Synchronized php-1.24wmf8/extensions/CirrusSearch/: SWAT - Fix Cirrus Special:Random (duration: 00m 04s) |
[production] |
| 15:13 |
<manybubbles> |
Synchronized php-1.24wmf9/extensions/CirrusSearch/: SWAT - Fix Cirrus Special:Random (duration: 00m 04s) |
[production] |
| 15:02 |
<manybubbles> |
Synchronized wmf-config/InitialiseSettings.php: SWAT - lower event logging rate for mediaviewer (duration: 00m 05s) |
[production] |
| 13:51 |
<_joe_> |
production puppet masters upgraded to puppet 3 |
[production] |
| 07:12 |
<springle> |
starting updateCollation on s3 frwikinews from tin |
[production] |
| 07:07 |
<springle> |
Synchronized wmf-config/InitialiseSettings.php: $wgCategoryCollation to uca-fr on frwikinews (duration: 00m 07s) |
[production] |
| 03:20 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Tue Jun 17 03:19:12 UTC 2014 (duration 19m 11s) |
[production] |
| 02:35 |
<LocalisationUpdate> |
completed (1.24wmf9) at 2014-06-17 02:34:09+00:00 |
[production] |
| 02:23 |
<LocalisationUpdate> |
completed (1.24wmf8) at 2014-06-17 02:22:46+00:00 |
[production] |
|
2014-06-16
§
|
| 23:12 |
<maxsem> |
Synchronized php-1.24wmf8/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/#/c/139562/ (duration: 00m 05s) |
[production] |
| 23:11 |
<maxsem> |
Synchronized php-1.24wmf9/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/#/c/139562/ (duration: 00m 06s) |
[production] |
| 23:05 |
<maxsem> |
Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/139888/ (duration: 00m 08s) |
[production] |
| 21:30 |
<ori> |
upgraded eventlogging to 3012aad |
[production] |
| 20:45 |
<ori> |
updated eventlogging to b4b42effc6 |
[production] |
| 17:36 |
<csteipp> |
Synchronized php-1.24wmf8/extensions/EducationProgram/includes/api/ApiAddStudents.php: Bug66631 (duration: 00m 05s) |
[production] |
| 17:34 |
<csteipp> |
Synchronized php-1.24wmf9/extensions/EducationProgram/includes/api/ApiAddStudents.php: (no message) (duration: 00m 05s) |
[production] |
| 15:59 |
<godog> |
manually ran update-ubuntu-mirror on carbon, successful |
[production] |
| 15:57 |
<awight> |
updated crm from e52a4eb1bfab622f612dc84f687678fff1fdbc04 to 26460d6eaec26861661322df8e9f07a8b0519677 |
[production] |
| 15:30 |
<ottomata> |
reinstalling analytics1018 |
[production] |
| 13:38 |
<twkozlowski> |
_joe_ also working on recovering the list which was deleted by mistake |
[production] |
| 13:37 |
<_joe_> |
closed wikimedia-de-by list |
[production] |
| 13:13 |
<_joe_> |
removing chip-l mailing list as for bug #63877 |
[production] |
| 13:03 |
<godog> |
restarting swift-proxy-server on ms-fe1001 to test statsd metrics |
[production] |
| 10:47 |
<godog> |
restarting swift-proxy-server on ms-fe3002 to test statsd metrics |
[production] |
| 10:23 |
<hoo> |
Touched all 1.24wmf8 extension/wikidata files and ran sync-common after that on mw1070 |
[production] |
| 10:18 |
<hoo> |
Synchronized php-1.24wmf8/extensions/Wikidata/: Update Wikidata to fix a suggester bug (duration: 00m 09s) |
[production] |
| 10:16 |
<godog> |
restarting swift-proxy-server on ms-fe3001 to test statsd metrics |
[production] |
| 10:13 |
<hoo> |
Synchronized php-1.24wmf9/extensions/Wikidata/: Update Wikidata to fix a suggester bug (duration: 00m 13s) |
[production] |
| 09:29 |
<apergos> |
restarted search1015 about 15 mns ago, it's now recovered afaict, restarted search1016, it's doing index setup now |
[production] |
| 03:00 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Jun 16 02:59:43 UTC 2014 (duration 59m 42s) |
[production] |
| 02:27 |
<LocalisationUpdate> |
completed (1.24wmf9) at 2014-06-16 02:26:05+00:00 |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.24wmf8) at 2014-06-16 02:14:38+00:00 |
[production] |