|
2017-06-08
ยง
|
| 18:55 |
<maxsem@tin> |
scap failed: average error rate on 1/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/3888cca979647b9381a7739b0bdbc88e for details) |
[production] |
| 18:49 |
<urandom> |
Restarting Cassandra, restbase-dev1001-a to test alternative disk access mode |
[production] |
| 18:42 |
<mutante> |
built gerrit_2.13.8+git1-wmf.5 on copper (T158946) |
[production] |
| 18:40 |
<maxsem@tin> |
Synchronized php-1.30.0-wmf.4/extensions/LoginNotify/: https://gerrit.wikimedia.org/r/#/c/357743/ (duration: 00m 44s) |
[production] |
| 18:36 |
<maxsem@tin> |
Synchronized php-1.30.0-wmf.4/includes/EditPage.php: https://gerrit.wikimedia.org/r/#/c/357855/ (duration: 00m 45s) |
[production] |
| 18:25 |
<maxsem@tin> |
Synchronized multiversion/submodules.json: https://gerrit.wikimedia.org/r/#/c/352985/3 (duration: 00m 43s) |
[production] |
| 18:17 |
<maxsem@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/356881/4 (duration: 00m 44s) |
[production] |
| 18:09 |
<maxsem@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/354731/6 (duration: 00m 44s) |
[production] |
| 17:55 |
<arlolra> |
Updated Parsoid to 108eed81 (T136653, T167081) |
[production] |
| 17:46 |
<arlolra@tin> |
Finished deploy [parsoid/deploy@f82cb4f]: Updating Parsoid to 108eed81 (duration: 10m 12s) |
[production] |
| 17:36 |
<arlolra@tin> |
Started deploy [parsoid/deploy@f82cb4f]: Updating Parsoid to 108eed81 |
[production] |
| 16:44 |
<nuria@tin> |
Finished deploy [analytics/refinery@2fbed63]: (no justification provided) (duration: 04m 08s) |
[production] |
| 16:40 |
<nuria@tin> |
Started deploy [analytics/refinery@2fbed63]: (no justification provided) |
[production] |
| 16:33 |
<godog> |
delete net.ifnames for ms-be2001 and ms-be2013 - T158429 |
[production] |
| 16:24 |
<bblack> |
cp1074: varnish-backend-restart for mailbox lag |
[production] |
| 15:22 |
<moritzm> |
updating mw1262-mw1265 to HHVM 3.18.2+wmf5 |
[production] |
| 15:11 |
<XioNoX> |
Upgrading rancid to 3 - T167288 |
[production] |
| 14:56 |
<moritzm> |
updating mw1261 to HHVM 3.18.2+wmf5 |
[production] |
| 14:54 |
<XioNoX> |
2 blackhole IPs pushed to cr* routers |
[production] |
| 14:02 |
<aude@tin> |
Synchronized wmf-config/InitialiseSettings-labs.php: Do not enable Wikibase data access yet on beta wiktionary (duration: 00m 43s) |
[production] |
| 13:47 |
<aude@tin> |
Synchronized php-1.30.0-wmf.4/extensions/RevisionSlider: Fix fatal error: T167359 (duration: 00m 44s) |
[production] |
| 13:41 |
<aude@tin> |
scap failed: average error rate on 1/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/3888cca979647b9381a7739b0bdbc88e for details) |
[production] |
| 13:33 |
<aude@tin> |
Synchronized php-1.30.0-wmf.4/extensions/Wikidata: Fix warning in date formatting T167360 (duration: 02m 16s) |
[production] |
| 13:31 |
<XioNoX> |
blackhole v4 IPs removed from all cr* routers |
[production] |
| 12:39 |
<moritzm> |
updating mwdebug* to HHVM 3.18.2+wmf5 |
[production] |
| 12:17 |
<moritzm> |
uploaded hhvm 3.18.2-dfsg-1+wmf5 to apt.wikimedia.org |
[production] |
| 12:17 |
<moritzm> |
updated hhvm 3.18.2-dfsg-1+wmf5 to apt.wikimedia.org |
[production] |
| 11:41 |
<marostegui> |
Drop table updates on s7 - T139342 |
[production] |
| 11:41 |
<moritzm> |
powercycling mw1294, mgmt is unresponsive |
[production] |
| 09:41 |
<moritzm> |
updating mysql-connector-java on hadoop cluster |
[production] |
| 09:05 |
<elukey> |
upgrade zookeeper packages to 3.4.5+dfsg-2+deb8u2 on conf100[123], conf200[23] and druid100[123] |
[production] |
| 08:58 |
<godog> |
swift eqiad-prod eqiad-prod: decom ms-be1005/6/7 - T166489 |
[production] |
| 08:50 |
<TabbyCat> |
Rename user "Mlpearc" to "FlightTime" on Central Auth is now finished (T166028) |
[production] |
| 08:36 |
<godog> |
temporarily stop ircecho on tegmen, puppet spam |
[production] |
| 08:21 |
<TabbyCat> |
Starting big global rename as requested in T166028 |
[production] |
| 07:00 |
<marostegui> |
Drop table updates on s6 - T139342 |
[production] |
| 05:59 |
<_joe_> |
uploading new service-checker version to reprepro, T167048 |
[production] |
| 05:54 |
<marostegui> |
Deploy alter table s2 - db1074 - T166205 |
[production] |
| 05:52 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1074 - T166205 (duration: 00m 43s) |
[production] |
| 05:43 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1076 - T166205 (duration: 00m 45s) |
[production] |
| 02:56 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Thu Jun 8 02:56:27 UTC 2017 (duration 6m 26s) |
[production] |
| 02:50 |
<l10nupdate@tin> |
scap sync-l10n completed (1.30.0-wmf.4) (duration: 05m 07s) |
[production] |
| 02:40 |
<twentyafterfour> |
deploying hotfix for T166958 |
[production] |
| 02:34 |
<l10nupdate@tin> |
scap sync-l10n completed (1.30.0-wmf.2) (duration: 08m 41s) |
[production] |
| 01:45 |
<mutante> |
manually running mediawiki maintenance job "echo_mail_batch" (on terbium as www-data, just like cron). did _NOT_ get denied by DB (T167373) |
[production] |
| 01:37 |
<maxsem@tin> |
Synchronized php-1.30.0-wmf.2/extensions/GeoData/includes/Searcher.php: Livehack to stop exceptions (duration: 00m 46s) |
[production] |
| 00:54 |
<mutante> |
cp4019 - powercycled (same as others) | lvs1007 - sits at installer - waiting for IP to be configured (T150256) |
[production] |
| 00:47 |
<mutante> |
cp1059 - same thing - powercycle after failed boot after reimaging script |
[production] |
| 00:41 |
<mutante> |
cp4011 - like cp4010 - powercycling (host down, console sat at initramfs). it hat the "did not detect disk by uid" issue but boots normal after powercycle |
[production] |
| 00:34 |
<mutante> |
cp4020 - powercycling (host down, console sat at initramfs) |
[production] |