|
2016-11-18
§
|
| 08:33 |
<elukey> |
kafka1022 up and running with kafka* daemon masked and broken disk removed from fstab (we mount partitions in there using UUIDs) |
[production] |
| 08:21 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Depool db2049 - T150876 (duration: 00m 49s) |
[production] |
| 08:07 |
<_joe_> |
physical powercycle of kafka1022 (broken disk) |
[production] |
| 07:55 |
<_joe_> |
rebooting kafka1022, a shower of defunct processes, kafka refuses to startup again |
[production] |
| 07:43 |
<_joe_> |
restarting kafka on kafka1022, too many open files |
[production] |
| 07:24 |
<moritzm> |
installing openjdk-7 security updates on trusty systems |
[production] |
| 02:31 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Fri Nov 18 02:31:52 UTC 2016 (duration 4m 40s) |
[production] |
| 02:27 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.3) (duration: 09m 18s) |
[production] |
| 00:11 |
<reedy@tin> |
Synchronized wmf-config/CommonSettings.php: wfLoadExtension for numerous extensions (duration: 00m 48s) |
[production] |
| 00:10 |
<reedy@tin> |
Synchronized wmf-config/extension-list: More extensions to extension.json (duration: 00m 48s) |
[production] |
|
2016-11-17
§
|
| 23:42 |
<krinkle@tin> |
Synchronized php-1.29.0-wmf.3/resources/src/mediawiki/mediawiki.js: Ie21f5c: undo temp revert for metric observation (duration: 00m 54s) |
[production] |
| 23:22 |
<krinkle@tin> |
Synchronized php-1.29.0-wmf.3/resources/src/mediawiki/mediawiki.js: Ie21f5c: temp revert cache-eval for metric observation (duration: 00m 49s) |
[production] |
| 22:24 |
<ejegg> |
updated SmashPig from 1e895ba4468ac5a2e0812dd5719fd42eb2ec6515 to 3cbb42fb7138e780bd76a2f17c48983f8f882c57 |
[production] |
| 22:08 |
<mutante> |
contint1001 - Apache fix puppetized, re-enabeld |
[production] |
| 22:02 |
<ejegg|food> |
updated SmashPig from cac1a196b0e4805ff38671364228a89f8d053c5c to 1e895ba4468ac5a2e0812dd5719fd42eb2ec6515 |
[production] |
| 21:58 |
<mutante> |
doc.wm.org Apache config live-hack fixed, puppet patch coming up |
[production] |
| 21:02 |
<Pchelolo> |
update RESTBase to b9722ba7c |
[production] |
| 20:59 |
<Pchelolo> |
update RESTBase to b9722ba7c - canary on restbase1007 |
[production] |
| 20:55 |
<Pchelolo> |
update RESTBase to b9722ba7c - staging |
[production] |
| 20:26 |
<jynus> |
applying schema change on s1 (page) T69223 |
[production] |
| 20:07 |
<thcipriani@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.3 |
[production] |
| 20:03 |
<krinkle@tin> |
Synchronized php-1.29.0-wmf.3/extensions/WikimediaEvents/modules/ext.wikimediaEvents.visibilitychange.js: Ibd0935bef8f (duration: 00m 48s) |
[production] |
| 19:58 |
<ejegg> |
extended session timeouts on payments-wiki |
[production] |
| 19:25 |
<addshore@tin> |
Synchronized php-1.29.0-wmf.3/extensions/Wikidata: {{gerrit|322092}} T150948 Backporting fix for quantity precision issue. (duration: 02m 17s) |
[production] |
| 19:07 |
<krenair@tin> |
Synchronized wmf-config/interwiki.php: https://gerrit.wikimedia.org/r/#/c/322141/ (duration: 00m 50s) |
[production] |
| 18:57 |
<bearND> |
deployed mobileapps bf44547 |
[production] |
| 18:53 |
<bearND> |
starting mobileapps deploy |
[production] |
| 18:36 |
<twentyafterfour> |
phabricator: deploy upstream fix for T150971 (upstream sha1: 7ebc47d906fe ) |
[production] |
| 18:16 |
<twentyafterfour> |
deploy more phabricator hotfixes |
[production] |
| 17:46 |
<twentyafterfour> |
unbreak search |
[production] |
| 17:40 |
<twentyafterfour> |
phabricator: deploying hotfix for T150965 |
[production] |
| 17:34 |
<godog> |
bounce gerrit on cobalt after https://gerrit.wikimedia.org/r/321398 |
[production] |
| 17:11 |
<twentyafterfour> |
restart apache2 on iridium to clear lagged queries refs T150965 |
[production] |
| 17:07 |
<ejegg> |
increased paypal job runner message limit to 5000 |
[production] |
| 17:02 |
<cmjohnson> |
powering down ms-be1016 to reseat the raid card. |
[production] |
| 16:52 |
<mobrovac> |
change-prop deploying ed3711b |
[production] |
| 16:51 |
<demon@tin> |
Synchronized docroot/: Unifying most docroots (duration: 00m 50s) |
[production] |
| 16:45 |
<demon@tin> |
Synchronized w/static.php: code duplication stuffs (duration: 00m 49s) |
[production] |
| 16:40 |
<godog> |
upgrade nginx on prometheus and thumbor machines |
[production] |
| 16:28 |
<Reedy> |
Deleted centralauth.oathauth_users row for Horst |
[production] |
| 16:14 |
<moritzm> |
restarting app server canaries to pick up libxslt update |
[production] |
| 15:52 |
<ema> |
rebooting primary LVS hosts for kernel updates |
[production] |
| 15:44 |
<ori@tin> |
Synchronized php-1.29.0-wmf.3/extensions/NavigationTiming/modules/ext.navigationTiming.js: I8e8ec96f: Don't report stats when page visibility changes during page load (duration: 00m 48s) |
[production] |
| 15:42 |
<ori@tin> |
Synchronized php-1.29.0-wmf.2/extensions/NavigationTiming/modules/ext.navigationTiming.js: I8e8ec96f: Dont report stats when page visibility changes during page load ; scap sync-file php-1.29.0-wmf.3/extensions/NavigationTiming/modules/ext.navigationTiming.js I8e8ec96f: Dont report stats when page visibility changes during page load (duration: 00m 51s) |
[production] |
| 15:24 |
<jynus> |
applying schema change on s4 (page) T69223 |
[production] |
| 15:10 |
<moritzm> |
installing libxslt security updates |
[production] |
| 15:02 |
<ema> |
rebooting secondary (inactive) LVS hosts for kernel updates |
[production] |
| 12:50 |
<moritzm> |
upgrading imagemagick on remaining image scalers (T141739) |
[production] |
| 10:47 |
<moritzm> |
installing trusty kernel updates |
[production] |
| 10:22 |
<elukey> |
cleanup on analytics1027 - Removed mysql-server-5.5 (not used) and ran apt autoremove (old kernels) |
[production] |