|
2016-11-28
§
|
| 19:23 |
<urandom> |
Stupidly issued iptables -F on restbase2012 <facepalm /> |
[production] |
| 18:08 |
<gehel> |
deploying latest WDQS GUI and Blazegraph |
[production] |
| 17:20 |
<_joe_> |
turned on the second commonswiki htmlCacheUpdate dedicated jobrunner (T151196) |
[production] |
| 16:50 |
<_joe_> |
upgrading HHVM on canaries |
[production] |
| 16:49 |
<_joe_> |
re-started the commonswiki htmlCacheUpdate dedicated jobrunner (T151196) |
[production] |
| 16:35 |
<addshore> |
RevisionSlider updates window finished 35 mins late! |
[production] |
| 16:34 |
<addshore@tin> |
Finished scap: RevisionSlider updates - [[gerrit:323384]], [[gerrit:323520]], [[gerrit:323521]], [[gerrit:323808]] (duration: 55m 25s) |
[production] |
| 15:39 |
<addshore@tin> |
Started scap: RevisionSlider updates - [[gerrit:323384]], [[gerrit:323520]], [[gerrit:323521]], [[gerrit:323808]] |
[production] |
| 15:36 |
<addshore> |
Turned change by ori to local commit on tin in core on .3 branch |
[production] |
| 15:35 |
<jynus> |
restarting db1069 (sanitarium) instances to apply new replication filters T151752 |
[production] |
| 15:13 |
<addshore> |
fixed /srv/mediawiki-staging/php-1.29.0-wmf.3 clone of mw core on tin due to T151676 |
[production] |
| 14:53 |
<zeljkof> |
EU SWAT finished |
[production] |
| 14:52 |
<zfilipin@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:323833|Allow sysop to revoke users from some groups on ne.wikipedia (T148171)]] (duration: 00m 45s) |
[production] |
| 14:37 |
<zfilipin@tin> |
Synchronized wmf-config/throttle.php: SWAT: [[gerrit:323557|[throttle] Remove old throttle rules]] (duration: 00m 44s) |
[production] |
| 14:33 |
<zfilipin@tin> |
Synchronized wmf-config/throttle.php: SWAT: [[gerrit:323555|[throttle] Exception for #MOWomenOnWikipedia Edit-A-Thon (T151650)]] (duration: 00m 45s) |
[production] |
| 14:33 |
<chasemp> |
labsdb1001 maintain-views --databases testwiki enwikivoyage enwiki --table page_assessments_projects --debug |
[production] |
| 14:31 |
<chasemp> |
labsdb1001 maintain-views --databases testwiki enwikivoyage enwiki --table page_assessments --debug |
[production] |
| 14:19 |
<zfilipin@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:322247|HD logos for multiple wikis (T150618)]] (duration: 00m 49s) |
[production] |
| 14:18 |
<zfilipin@tin> |
Synchronized static/images/project-logos/: SWAT: [[gerrit:322247|HD logos for multiple wikis (T150618)]] (duration: 00m 47s) |
[production] |
| 13:01 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2049 - T150876 (duration: 00m 45s) |
[production] |
| 12:35 |
<marostegui> |
Stop replication db1052 (depooled) - maintenance - T150802 |
[production] |
| 12:17 |
<hoo> |
Killed the Wikidata json dumpers on snapshot1007 due to T151356. Will be restarted once a fix has been deployed. |
[production] |
| 10:34 |
<volans> |
fixed permissions of old /var/log/hhvm/error.log-20160829 on osmium |
[production] |
| 10:19 |
<volans> |
fixed permissions of files in /srv/mediawiki-staging on tin and mira |
[production] |
| 10:18 |
<_joe_> |
stopping the dedicated commonswiki htmlCacheUpdate job runner, T151196 |
[production] |
| 09:09 |
<marostegui> |
Deploy ALTER table (add an index) db1040 (master) commonswiki.revision - T147305 |
[production] |
| 07:42 |
<marostegui> |
Deploying alter table s5 dewiki.revision - T148967 |
[production] |
| 07:38 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1092 - T151272 (duration: 00m 47s) |
[production] |
| 07:18 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Added comments to db1044 status - T150802 (duration: 00m 45s) |
[production] |
| 07:08 |
<marostegui> |
Stop MySQl on db1095 - maintenance T150802 |
[production] |
| 07:03 |
<marostegui> |
Stop MySQL on db1044 - (depooled) maintenance - T150802 |
[production] |
| 02:05 |
<Reedy> |
fixed localisationupdate clone of mw core on tin due to T151676 |
[production] |
| 02:00 |
<l10nupdate@tin> |
LocalisationUpdate failed: git pull of core failed |
[production] |
|
2016-11-27
§
|
| 21:47 |
<legoktm> |
created wmf/1.29.0-wmf.3 branch pointing at master for mediawiki/extensions/ElectronPdfService to workaround T151725 |
[production] |
| 09:35 |
<elukey> |
removed all the files not used in /tmp on stat1002 after a follow up with the owner |
[production] |
| 06:20 |
<ori@tin> |
Synchronized php-1.29.0-wmf.3/api.php: Bandaid: make API reqs fail fast if User-Agent ~= Parsoid and Host ~= eu.wikipedia.org (duration: 00m 50s) |
[production] |
| 05:36 |
<ori> |
Commented-out lived-hack from mw1290; if we see memory growth now, Parsoid would be strongly implicated. |
[production] |
| 05:33 |
<ori> |
With Parsoid requests hacked to fail fast, mw1290 is not showing the kind of aggressive growth in memory usage we're seeing on other API servers |
[production] |
| 05:30 |
<godog> |
roll restarting hhvm across api_cluster when hhvm uses more than 40% of memory |
[production] |
| 05:21 |
<ori> |
Live-hacked api.php on mw1290 to die if request user-agent contains 'Parsoid'; restarted HHVM. |
[production] |
| 05:17 |
<godog> |
roll restarting hhvm across api_cluster when hhvm uses more than 40% of memory |
[production] |
| 04:57 |
<godog> |
roll-restart hhvm on api_appcluster for on machines with hhvm leaking memory |
[production] |
| 03:22 |
<godog> |
roll-restart hhvm across api_appserver |
[production] |
| 02:41 |
<godog> |
dumping hhvm backtraces and roll-restart on affected api machines |
[production] |
| 02:00 |
<l10nupdate@tin> |
LocalisationUpdate failed: git pull of core failed |
[production] |