|
2016-06-17
§
|
| 09:31 |
<_joe_> |
powercycling mw1140, OOMd |
[production] |
| 09:30 |
<moritzm> |
rolling reboot of mw1153,mw1155,mw1156 into new kernels |
[production] |
| 08:29 |
<hashar> |
Restarting Jenkins on gallium. Web interface at least is deadlocked somehow |
[production] |
| 07:23 |
<jynus> |
backuping and reimaging db1072 |
[production] |
| 07:18 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1072 for maintenance (duration: 00m 31s) |
[production] |
| 07:11 |
<mobrovac> |
restbase started mobile-sections dump on restbase1009 for T136964 |
[production] |
| 07:02 |
<mobrovac> |
change-prop restarting it to apply https://gerrit.wikimedia.org/r/294880 |
[production] |
| 06:40 |
<moritzm> |
installing apache update on palladium |
[production] |
| 06:16 |
<akosiaris> |
_joe_ restarted zotero on sca1001 |
[production] |
| 06:16 |
<akosiaris> |
restarted zotero on sca1002 |
[production] |
| 06:04 |
<root@palladium> |
conftool action : set/weight=25; selector: cluster=api_appserver,name=mw127.* |
[production] |
| 05:58 |
<root@palladium> |
conftool action : set/pooled=yes:weight=20; selector: cluster=api_appserver,name=mw127.* |
[production] |
| 02:31 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Fri Jun 17 02:31:00 UTC 2016 (duration 6m 26s) |
[production] |
| 02:24 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.6) (duration: 09m 46s) |
[production] |
|
2016-06-16
§
|
| 23:44 |
<ebernhardson@tin> |
Synchronized php-1.28.0-wmf.6/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T137167: TextCat A/B test for Language Identification (duration: 00m 25s) |
[production] |
| 23:24 |
<ebernhardson@tin> |
Synchronized php-1.28.0-wmf.6/extensions/WikimediaEvents/extension.json: T137167: TextCat A/B test for Language Identification (duration: 00m 24s) |
[production] |
| 23:19 |
<ebernhardson@tin> |
Synchronized php-1.28.0-wmf.6/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T137167: TextCat A/B test for Language Identification (duration: 00m 24s) |
[production] |
| 23:16 |
<ebernhardson@tin> |
Synchronized wmf-config/InitialiseSettings.php: T137167: search: Dependent config for textcat AB test. (duration: 00m 26s) |
[production] |
| 23:11 |
<ebernhardson@tin> |
Synchronized wmf-config/InitialiseSettings.php: T137888: Two permission changes at urwiki (duration: 00m 27s) |
[production] |
| 23:07 |
<ebernhardson@tin> |
Synchronized wmf-config/InitialiseSettings-labs.php: T127250: Prepare Wikidata descriptions on mobile for production rollout (duration: 00m 27s) |
[production] |
| 22:33 |
<maxsem@tin> |
Synchronized php-1.28.0-wmf.6/extensions/Kartographer: https://gerrit.wikimedia.org/r/294856 https://gerrit.wikimedia.org/r/294855 (duration: 00m 30s) |
[production] |
| 22:24 |
<maxsem@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/294854/ (duration: 00m 26s) |
[production] |
| 21:15 |
<hashar@tin> |
Synchronized php-1.28.0-wmf.6/extensions/VisualEditor/ApiVisualEditor.php: Pass empty summary to parseAndStash() to avoid warnings T137995 (duration: 00m 39s) |
[production] |
| 19:05 |
<hashar@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.6 |
[production] |
| 18:37 |
<tgr> |
running invalidateUserSessions.php for T137799 |
[production] |
| 18:22 |
<mobrovac> |
change-prop deploying bc87a1fecfa |
[production] |
| 16:36 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Set all new slaves to medium weight (300) after warm up (duration: 00m 25s) |
[production] |
| 15:37 |
<jynus> |
deleted sqldata.s6 from labsdb1008 - space issues caused by queries creating temporary tables |
[production] |
| 15:27 |
<thcipriani@tin> |
Synchronized php-1.28.0-wmf.6/extensions/ORES/includes/Hooks.php: SWAT: [[gerrit:294712|Performance boost on hidenondamaging]] (duration: 00m 35s) |
[production] |
| 15:23 |
<moritzm> |
rolling reboot of restbase1008 - restbase1011 for upgrade to Linux 4.4 |
[production] |
| 15:21 |
<thcipriani@tin> |
Synchronized php-1.28.0-wmf.6/extensions/ORES: SWAT: [[gerrit:294711|Skip when an edit is errored in PopulateDatabase.php]] (duration: 00m 30s) |
[production] |
| 15:04 |
<root@palladium> |
conftool action : set/pooled=yes; selector: name=mw1262.eqiad.wmnet |
[production] |
| 14:31 |
<twentyafterfour> |
re-enabled and ran puppet agent --test on iridium. Everything appears to be normal. |
[production] |
| 13:04 |
<mobrovac> |
scb1001 enabled puppet back |
[production] |
| 12:57 |
<gehel> |
rebalancing shards on elasticsearch equiad cluster |
[production] |
| 12:33 |
<Amir1> |
manually restarted celery-ores-worker in scb1001 |
[production] |
| 12:32 |
<moritzm> |
installing apache2 trusty update on graphite1001 |
[production] |
| 12:32 |
<Amir1> |
manually restarted celery-ores-worker in scb1002 |
[production] |
| 12:10 |
<moritzm> |
restarted hhvm on mw1137, got stuck |
[production] |
| 10:44 |
<moritzm> |
depooling mw1154 for kernel update/reboot |
[production] |
| 10:14 |
<mobrovac> |
scb1001 disabling puppet for a while to manually test changeprop with transclusion rules |
[production] |
| 09:59 |
<mobrovac> |
restbase deploy end of ebeaa46 |
[production] |
| 09:56 |
<_joe_> |
powercycling mw1143, unresponsive on ssh, console |
[production] |
| 09:48 |
<mobrovac> |
restbase deploy start of ebeaa46 |
[production] |
| 09:18 |
<hashar@tin> |
Synchronized php-1.28.0-wmf.6/extensions/MobileFrontend: MobileFrontend RL registration issue preventing Special:Nearby from working properly T137919 (duration: 00m 36s) |
[production] |
| 08:41 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Pool db1085, increase weight of all new db servers (duration: 00m 29s) |
[production] |
| 08:15 |
<jynus> |
rebooting db1085 before putting it back into production |
[production] |
| 02:34 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.5) (duration: 15m 49s) |
[production] |
| 00:57 |
<twentyafterfour> |
puppet disabled on iridium because https://gerrit.wikimedia.org/r/#/c/294653/ needs to merge (hotfix in preamble.php which puppet will undo if it's allowed to run) |
[production] |
| 00:43 |
<twentyafterfour> |
phabricator upgrade/maintenance complete. Everything appears to be back up and running normally. |
[production] |