|
2017-06-08
§
|
| 16:24 |
<bblack> |
cp1074: varnish-backend-restart for mailbox lag |
[production] |
| 15:22 |
<moritzm> |
updating mw1262-mw1265 to HHVM 3.18.2+wmf5 |
[production] |
| 15:11 |
<XioNoX> |
Upgrading rancid to 3 - T167288 |
[production] |
| 14:56 |
<moritzm> |
updating mw1261 to HHVM 3.18.2+wmf5 |
[production] |
| 14:54 |
<XioNoX> |
2 blackhole IPs pushed to cr* routers |
[production] |
| 14:02 |
<aude@tin> |
Synchronized wmf-config/InitialiseSettings-labs.php: Do not enable Wikibase data access yet on beta wiktionary (duration: 00m 43s) |
[production] |
| 13:47 |
<aude@tin> |
Synchronized php-1.30.0-wmf.4/extensions/RevisionSlider: Fix fatal error: T167359 (duration: 00m 44s) |
[production] |
| 13:41 |
<aude@tin> |
scap failed: average error rate on 1/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/3888cca979647b9381a7739b0bdbc88e for details) |
[production] |
| 13:33 |
<aude@tin> |
Synchronized php-1.30.0-wmf.4/extensions/Wikidata: Fix warning in date formatting T167360 (duration: 02m 16s) |
[production] |
| 13:31 |
<XioNoX> |
blackhole v4 IPs removed from all cr* routers |
[production] |
| 12:39 |
<moritzm> |
updating mwdebug* to HHVM 3.18.2+wmf5 |
[production] |
| 12:17 |
<moritzm> |
uploaded hhvm 3.18.2-dfsg-1+wmf5 to apt.wikimedia.org |
[production] |
| 12:17 |
<moritzm> |
updated hhvm 3.18.2-dfsg-1+wmf5 to apt.wikimedia.org |
[production] |
| 11:41 |
<marostegui> |
Drop table updates on s7 - T139342 |
[production] |
| 11:41 |
<moritzm> |
powercycling mw1294, mgmt is unresponsive |
[production] |
| 09:41 |
<moritzm> |
updating mysql-connector-java on hadoop cluster |
[production] |
| 09:05 |
<elukey> |
upgrade zookeeper packages to 3.4.5+dfsg-2+deb8u2 on conf100[123], conf200[23] and druid100[123] |
[production] |
| 08:58 |
<godog> |
swift eqiad-prod eqiad-prod: decom ms-be1005/6/7 - T166489 |
[production] |
| 08:50 |
<TabbyCat> |
Rename user "Mlpearc" to "FlightTime" on Central Auth is now finished (T166028) |
[production] |
| 08:36 |
<godog> |
temporarily stop ircecho on tegmen, puppet spam |
[production] |
| 08:21 |
<TabbyCat> |
Starting big global rename as requested in T166028 |
[production] |
| 07:00 |
<marostegui> |
Drop table updates on s6 - T139342 |
[production] |
| 05:59 |
<_joe_> |
uploading new service-checker version to reprepro, T167048 |
[production] |
| 05:54 |
<marostegui> |
Deploy alter table s2 - db1074 - T166205 |
[production] |
| 05:52 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1074 - T166205 (duration: 00m 43s) |
[production] |
| 05:43 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1076 - T166205 (duration: 00m 45s) |
[production] |
| 02:56 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Thu Jun 8 02:56:27 UTC 2017 (duration 6m 26s) |
[production] |
| 02:50 |
<l10nupdate@tin> |
scap sync-l10n completed (1.30.0-wmf.4) (duration: 05m 07s) |
[production] |
| 02:40 |
<twentyafterfour> |
deploying hotfix for T166958 |
[production] |
| 02:34 |
<l10nupdate@tin> |
scap sync-l10n completed (1.30.0-wmf.2) (duration: 08m 41s) |
[production] |
| 01:45 |
<mutante> |
manually running mediawiki maintenance job "echo_mail_batch" (on terbium as www-data, just like cron). did _NOT_ get denied by DB (T167373) |
[production] |
| 01:37 |
<maxsem@tin> |
Synchronized php-1.30.0-wmf.2/extensions/GeoData/includes/Searcher.php: Livehack to stop exceptions (duration: 00m 46s) |
[production] |
| 00:54 |
<mutante> |
cp4019 - powercycled (same as others) | lvs1007 - sits at installer - waiting for IP to be configured (T150256) |
[production] |
| 00:47 |
<mutante> |
cp1059 - same thing - powercycle after failed boot after reimaging script |
[production] |
| 00:41 |
<mutante> |
cp4011 - like cp4010 - powercycling (host down, console sat at initramfs). it hat the "did not detect disk by uid" issue but boots normal after powercycle |
[production] |
| 00:34 |
<mutante> |
cp4020 - powercycling (host down, console sat at initramfs) |
[production] |
| 00:31 |
<mutante> |
cp2012 - fixed salt key issue as for cp3005 (delete key, stop/start minion, accept new key) |
[production] |
| 00:25 |
<mutante> |
salt-master: deleted salt-key for cp3005, stopped started minion cp3005 - key got accepted again (was: Salt Master has rejected this minion's public key) |
[production] |
|
2017-06-07
§
|
| 23:33 |
<ppchelko@tin> |
Finished deploy [trending-edits/deploy@e0a8716]: Include reverts from bots to get rid of false positives (duration: 07m 00s) |
[production] |
| 23:30 |
<catrope@tin> |
Synchronized php-1.30.0-wmf.4/extensions/RelatedArticles/resources/ext.relatedArticles.readMore.eventLogging/index.js: T167236 (duration: 00m 43s) |
[production] |
| 23:28 |
<catrope@tin> |
Synchronized wmf-config/InitialiseSettings.php: Relaunch related pages A/B test to 98% of users on enwiki (T167310) (duration: 00m 44s) |
[production] |
| 23:26 |
<ppchelko@tin> |
Started deploy [trending-edits/deploy@e0a8716]: Include reverts from bots to get rid of false positives |
[production] |
| 22:24 |
<bblack> |
reimaging ex-cache_maps hosts (fresh role::spare::system installs) |
[production] |
| 22:18 |
<bblack> |
puppet node clean+deactivate for cp3003 |
[production] |
| 22:15 |
<bblack> |
lvs4002 - restarting pybal to remove old maps table entries |
[production] |
| 22:14 |
<bblack> |
lvs3002 - restarting pybal to remove old maps table entries |
[production] |
| 22:13 |
<bblack> |
lvs2002 - restarting pybal to remove old maps table entries |
[production] |
| 22:13 |
<bblack> |
lvs1002 - restarting pybal to remove old maps table entries |
[production] |
| 22:12 |
<bblack> |
lvs4004 - restarting pybal to remove old maps table entries |
[production] |
| 22:11 |
<bblack> |
lvs3004 - restarting pybal to remove old maps table entries |
[production] |