|
2014-06-26
§
|
| 10:52 |
<godog> |
stopping swift on ms-be3003 |
[production] |
| 10:12 |
<godog> |
upgrading ms-be3001 to swift icehouse |
[production] |
| 06:26 |
<springle> |
ran operations/software maintain-replicas.pl and fedtables.pl on labsdbs for bug 59683 |
[production] |
| 05:54 |
<Tim> |
on mw1014: reformatted the /tmp partition |
[production] |
| 05:50 |
<Tim> |
on mw1014: stopped job runner due to bad /tmp |
[production] |
| 04:45 |
<ori> |
mw1014 is sad, has filesystem issues: "Attempt to read block from filesystem resulted in short read while trying to open /tmp". Puppet can't run. Should be depooled. |
[production] |
| 03:34 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Jun 26 03:33:19 UTC 2014 (duration 33m 18s) |
[production] |
| 03:02 |
<LocalisationUpdate> |
completed (1.24wmf10) at 2014-06-26 03:01:43+00:00 |
[production] |
| 02:33 |
<LocalisationUpdate> |
completed (1.24wmf9) at 2014-06-26 02:31:50+00:00 |
[production] |
|
2014-06-25
§
|
| 23:43 |
<awight> |
updated crm from f3389daa94e9ad924175bdf0d5bc09c4a26aeb8c to e2fe03a9cd51e30206d9a1114d62dfbd6960816b |
[production] |
| 23:27 |
<catrope> |
Finished scap: Updating Wikidata and TimedMediaHandler (duration: 04m 24s) |
[production] |
| 23:23 |
<catrope> |
Started scap: Updating Wikidata and TimedMediaHandler |
[production] |
| 21:22 |
<hashar> |
puppet fixed on gallium / lanthanum . It was missing a group definition. All fixed! Thanks Chase. |
[production] |
| 20:53 |
<hashar> |
puppet broken on gallium.wikimedia.org and lanthanum.eqiad.wmnet . That is being looked at. |
[production] |
| 20:34 |
<subbu> |
deployed parsoid 4ef9d6be |
[production] |
| 19:38 |
<manybubbles> |
restarted Cirrus scripts after incident - the index rebuilds had to be completely restarted - sanity checking was simply paused |
[production] |
| 18:54 |
<yurik> |
Synchronized wmf-config/PrivateSettings.php: Removed obsolete ZRMA user/pswd (duration: 01m 06s) |
[production] |
| 18:46 |
<yurik> |
Finished scap: Removing ZeroRatedMobileAccess ext settings, depl latest JsonConfig/ZeroBanner/Portal (duration: 29m 09s) |
[production] |
| 18:17 |
<yurik> |
Started scap: Removing ZeroRatedMobileAccess ext settings, depl latest JsonConfig/ZeroBanner/Portal |
[production] |
| 17:41 |
<demon> |
Synchronized wmf-config/: Cirrus back on for wikis that had it before. Back to square 1 (duration: 00m 04s) |
[production] |
| 17:29 |
<mwalker> |
updating fundraising tools from 5f3a7316b636c0723ce3fa353186d4041b662872 to cdc4b73bd59d27c8d386b6df629b1c574cfed85f |
[production] |
| 17:06 |
<manybubbles> |
success! |
[production] |
| 17:06 |
<manybubbles> |
Synchronized wmf-config/: try to fix cirrus (duration: 00m 04s) |
[production] |
| 16:51 |
<andrewbogott> |
restarted apache on palladium -- _that_ helped |
[production] |
| 16:49 |
<andrewbogott> |
it didn't help |
[production] |
| 16:49 |
<andrewbogott> |
restarting puppetmaster on palladium |
[production] |
| 16:42 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: Disable Cirrus everywhere but testwiki (duration: 00m 04s) |
[production] |
| 16:23 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: Roll back previous Cirrus deploy (duration: 00m 05s) |
[production] |
| 16:23 |
<demon> |
Synchronized wmf-config/CommonSettings.php: Roll back previous Cirrus deploy (duration: 00m 04s) |
[production] |
| 16:16 |
<demon> |
Synchronized wmf-config/CommonSettings.php: I4c54357a: most remaining wikis getting Cirrus as primary (duration: 00m 04s) |
[production] |
| 16:16 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: I4c54357a: most remaining wikis getting Cirrus as primary (duration: 00m 04s) |
[production] |
| 15:27 |
<manybubbles> |
Synchronized php-1.24wmf9/extensions/Wikidata/: SWAT - fix rtl issue (duration: 00m 10s) |
[production] |
| 15:23 |
<ottomata> |
reinstalling elastic1017,1018,1019 |
[production] |
| 15:20 |
<manybubbles> |
Synchronized php-1.24wmf10/extensions/Wikidata/: SWAT - fix rtl issue (duration: 00m 12s) |
[production] |
| 14:10 |
<Krinkle> |
Upgrade npm from v1.4.5 to v1.4.16 on integration-slave1001 and integration-slave1002 |
[production] |
| 14:10 |
<Krinkle> |
Upgraded npm from v1.4.13 to v1.4.16 on integration-slave1003 to fix https://github.com/npm/npm/issues/5472 and repooling |
[production] |
| 13:30 |
<Krinkle> |
Depooling integration-slave1003 as almost every other -npm build on this node fails due to corrupted ~/.npm cache |
[production] |
| 12:52 |
<manybubbles> |
cirrus rebuild update: starting from mediawiki reindex step for all alphabetical wikis that have finished so far |
[production] |
| 12:25 |
<hashar> |
Upgraded Zuul 9839edb..b7fc126 Brings patchset 20 of Zuul cloner ( https://review.openstack.org/#/c/70373/ ) |
[production] |
| 12:02 |
<akosiaris> |
upgraded etherpad.wikimedia.org to etherpad-lite 1.4.0 |
[production] |
| 11:12 |
<paravoid> |
switching inbound email for wikimedia.org to polonium/mchenry |
[production] |
| 10:35 |
<_joe_> |
restarted lucene on search1016 as it was stuck there as well, once search1015 is up and running |
[production] |
| 10:06 |
<_joe_> |
restarted lucene on search1015, it was stuck |
[production] |
| 07:37 |
<springle> |
Synchronized wmf-config/db-eqiad.php: incremental LB bump on db1009 and db1021 traffic samplers (duration: 00m 07s) |
[production] |
| 06:31 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1021 with traffic sampling (duration: 00m 09s) |
[production] |
| 06:01 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1021, db1049 to normal load (duration: 00m 07s) |
[production] |
| 05:05 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1049, warm up (duration: 00m 08s) |
[production] |
| 02:50 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Jun 25 02:48:53 UTC 2014 (duration 48m 52s) |
[production] |
| 02:39 |
<springle> |
xtrabackup clone db1005 to db1049 |
[production] |
| 02:27 |
<LocalisationUpdate> |
completed (1.24wmf10) at 2014-06-25 02:25:57+00:00 |
[production] |