|
2014-06-26
§
|
| 19:06 |
<RobH> |
blog is back online after a number of reboots due to raid rebuild issues |
[production] |
| 18:20 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf11 |
[production] |
| 18:16 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf10 |
[production] |
| 18:15 |
<reedy> |
Synchronized php-1.24wmf10/includes/api/ApiQueryRecentChanges.php: Id9c316733896a27ce3f6c3e0e5efdf62f7d1ff1b (duration: 00m 14s) |
[production] |
| 18:09 |
<ottomata> |
starting new elasticsearch nodes 1017,1018,1019 |
[production] |
| 18:05 |
<RobH> |
aware of holmium issue (old varnish), in process of repair, blog is down |
[production] |
| 17:05 |
<reedy> |
Synchronized php-1.24wmf11/resources/Resources.php: I1237909d7e058137d55e5de9fa4d64fe1f7f9472 (duration: 00m 14s) |
[production] |
| 17:04 |
<reedy> |
Finished scap: l10nupdate for 1.24wmf11 for Skins I9395b0e1983122b12bedf003d6398da5ddfd5651 (duration: 16m 35s) |
[production] |
| 16:48 |
<reedy> |
Started scap: l10nupdate for 1.24wmf11 for Skins I9395b0e1983122b12bedf003d6398da5ddfd5651 |
[production] |
| 16:46 |
<reedy> |
Purged l10n cache for 1.24wmf4 |
[production] |
| 16:45 |
<reedy> |
Purged l10n cache for 1.24wmf5 |
[production] |
| 16:45 |
<reedy> |
Purged l10n cache for 1.24wmf6 |
[production] |
| 16:45 |
<reedy> |
Purged l10n cache for 1.24wmf7 |
[production] |
| 16:44 |
<reedy> |
Purged l10n cache for 1.24wmf8 |
[production] |
| 16:32 |
<reedy> |
Finished scap: testwiki to 1.24wmf11 and build l10n cache (duration: 27m 20s) |
[production] |
| 16:05 |
<reedy> |
Started scap: testwiki to 1.24wmf11 and build l10n cache |
[production] |
| 16:00 |
<reedy> |
Started scap: testwiki to 1.24wmf11 and build l10n cache |
[production] |
| 15:55 |
<reedy> |
Started scap: testwiki to 1.24wmf11 and build l10n cache |
[production] |
| 15:54 |
<reedy> |
Started scap: testwiki to 1.24wmf11 and build l10n cache |
[production] |
| 15:24 |
<cmjohnson1> |
shutting down holmium to replace disk |
[production] |
| 14:35 |
<bblack> |
restarted nova-network on labnet1001 |
[production] |
| 14:26 |
<hashar> |
updated zuul cloner in git repo and deployed zuul ( tag is wmf-deploy-20140626-1 ) |
[production] |
| 13:54 |
<godog> |
remounted (broken) sdk1 on ms-be3003 |
[production] |
| 13:32 |
<cmjohnson1> |
powering down dataset1001 -relocating to 10G rack |
[production] |
| 13:26 |
<reedy> |
Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/142142/ No-op for labs (duration: 00m 16s) |
[production] |
| 12:55 |
<hashar> |
Jenkins: updates jobs for extensions (phpunit and qunit) to use the mw-run-update-script.sh instead of update.php . That runs update.php twice, the first time logging sql to a file that can be archived. {{gerrit|141851}} |
[production] |
| 12:48 |
<mark> |
Deactivated BGP session to AS13030 |
[production] |
| 11:01 |
<hashar> |
Replacing operations-puppet-validate job with operations-puppet-pplint-HEAD which is faster and can run concurrently on multiple boxes. {{gerrit|142223}} |
[production] |
| 10:52 |
<godog> |
stopping swift on ms-be3003 |
[production] |
| 10:12 |
<godog> |
upgrading ms-be3001 to swift icehouse |
[production] |
| 06:26 |
<springle> |
ran operations/software maintain-replicas.pl and fedtables.pl on labsdbs for bug 59683 |
[production] |
| 05:54 |
<Tim> |
on mw1014: reformatted the /tmp partition |
[production] |
| 05:50 |
<Tim> |
on mw1014: stopped job runner due to bad /tmp |
[production] |
| 04:45 |
<ori> |
mw1014 is sad, has filesystem issues: "Attempt to read block from filesystem resulted in short read while trying to open /tmp". Puppet can't run. Should be depooled. |
[production] |
| 03:34 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Jun 26 03:33:19 UTC 2014 (duration 33m 18s) |
[production] |
| 03:02 |
<LocalisationUpdate> |
completed (1.24wmf10) at 2014-06-26 03:01:43+00:00 |
[production] |
| 02:33 |
<LocalisationUpdate> |
completed (1.24wmf9) at 2014-06-26 02:31:50+00:00 |
[production] |
|
2014-06-25
§
|
| 23:43 |
<awight> |
updated crm from f3389daa94e9ad924175bdf0d5bc09c4a26aeb8c to e2fe03a9cd51e30206d9a1114d62dfbd6960816b |
[production] |
| 23:27 |
<catrope> |
Finished scap: Updating Wikidata and TimedMediaHandler (duration: 04m 24s) |
[production] |
| 23:23 |
<catrope> |
Started scap: Updating Wikidata and TimedMediaHandler |
[production] |
| 21:22 |
<hashar> |
puppet fixed on gallium / lanthanum . It was missing a group definition. All fixed! Thanks Chase. |
[production] |
| 20:53 |
<hashar> |
puppet broken on gallium.wikimedia.org and lanthanum.eqiad.wmnet . That is being looked at. |
[production] |
| 20:34 |
<subbu> |
deployed parsoid 4ef9d6be |
[production] |
| 19:38 |
<manybubbles> |
restarted Cirrus scripts after incident - the index rebuilds had to be completely restarted - sanity checking was simply paused |
[production] |
| 18:54 |
<yurik> |
Synchronized wmf-config/PrivateSettings.php: Removed obsolete ZRMA user/pswd (duration: 01m 06s) |
[production] |
| 18:46 |
<yurik> |
Finished scap: Removing ZeroRatedMobileAccess ext settings, depl latest JsonConfig/ZeroBanner/Portal (duration: 29m 09s) |
[production] |
| 18:17 |
<yurik> |
Started scap: Removing ZeroRatedMobileAccess ext settings, depl latest JsonConfig/ZeroBanner/Portal |
[production] |
| 17:41 |
<demon> |
Synchronized wmf-config/: Cirrus back on for wikis that had it before. Back to square 1 (duration: 00m 04s) |
[production] |
| 17:29 |
<mwalker> |
updating fundraising tools from 5f3a7316b636c0723ce3fa353186d4041b662872 to cdc4b73bd59d27c8d386b6df629b1c574cfed85f |
[production] |
| 17:06 |
<manybubbles> |
success! |
[production] |