|
2016-03-14
§
|
| 12:57 |
<elukey> |
removed kafka1001 from eventbut's pool via confd |
[production] |
| 11:47 |
<moritzm> |
installing security updates for openssl, curl, gcrypt, libpng, jasper, expat and libxml2 on remaining mw1* app servers (along with HHVM restarts) |
[production] |
| 10:56 |
<moritzm> |
installing rsync security updates |
[production] |
| 10:38 |
<godog> |
shut restbase100[12] |
[production] |
| 10:37 |
<moritzm> |
installing security updates for openssl, curl, gcrypt, libpng, jasper, expat and libxml2 on mw2* (along with HHVM restarts) |
[production] |
| 10:27 |
<mobrovac> |
restbase restarting restbase in prod to apply https://gerrit.wikimedia.org/r/276728 |
[production] |
| 10:13 |
<mobrovac> |
restbase deployed 3bedb8f on restbase101[01].eqiad.wmnet |
[production] |
| 10:06 |
<mobrovac> |
restbase stopping restbase on rb100[12] before full deprovisioning |
[production] |
| 09:46 |
<godog> |
depool restbase1001 / restbase1002 before deprovisioning |
[production] |
| 05:19 |
<ori> |
Restarting HHVM on mw1025 with known-bad config option 'hhvm.enable_reusable_tc = true' for debugging |
[production] |
| 02:38 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Mon Mar 14 02:38:54 UTC 2016 (duration 8m 41s) |
[production] |
| 02:30 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.16) (duration: 13m 23s) |
[production] |
|
2016-03-11
§
|
| 23:21 |
<krinkle@tin> |
Synchronized wmf-config/InitialiseSettings.php: test2wiki favicon (duration: 00m 30s) |
[production] |
| 22:50 |
<aaron@tin> |
Synchronized wmf-config/jobqueue-eqiad.php: Disable persistent redis connections and bump timeout a bit (duration: 00m 38s) |
[production] |
| 19:40 |
<jynus> |
restarting dbstore1002 |
[production] |
| 18:51 |
<mutante> |
mw1001 - removed font again, mw1153 confirmed puppet installs it (only) on imagescalers |
[production] |
| 18:42 |
<mutante> |
mw1001 install fonts-gujr-extra to confirm gerrit 276501 is fine |
[production] |
| 17:38 |
<elukey> |
Increased mediawiki::jobrunner::runners_basic to 30 for mw116[123789] |
[production] |
| 16:38 |
<mattflaschen@tin> |
Synchronized wmf-config/InitialiseSettings.php: Comment-only change (duration: 00m 41s) |
[production] |
| 15:44 |
<chasemp> |
labtestmetal and labtestvirt I am commondeering for some nfs and storage testing that cannot be done in labs, I will reimage when done |
[production] |
| 15:41 |
<elukey> |
forced puppet agent -tv on mw1164 |
[production] |
| 15:39 |
<elukey> |
increased mediawiki::jobrunner::runners_basic from 20 to 30 for mw116[3-6] |
[production] |
| 15:26 |
<gehel> |
deploying https://gerrit.wikimedia.org/r/#/c/274382/ - new defined type not used anywhere, should have less than zero impact. |
[production] |
| 15:14 |
<demon@tin> |
Synchronized multiversion/updateBranchPointers: (no message) (duration: 00m 25s) |
[production] |
| 15:12 |
<demon@tin> |
Synchronized wmf-config/throttle.php: Procomuns Viquimarató - Barcelona throttle rule (duration: 00m 31s) |
[production] |
| 15:06 |
<moritzm> |
installing security updates for openssl, curl, gcrypt, libpng, jasper, expat and libxml2 on mw1018-1025 (and restarts of HHVM) |
[production] |
| 15:04 |
<demon@tin> |
Synchronized README: no-op, co-master sync (duration: 00m 30s) |
[production] |
| 15:03 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: moving large.dblist back to wmf.16, did not help |
[production] |
| 14:45 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: moving largest.dblist back to wmf.15 for now |
[production] |
| 14:33 |
<godog> |
run swiftrepl thumbs eqiad -> codfw with concurrency 96 |
[production] |
| 14:25 |
<godog> |
run swiftrepl thumbs eqiad -> codfw with concurrency 128 |
[production] |
| 14:12 |
<moritzm> |
installed security updates for openssl, curl, gcrypt, libpng, jasper, expat and libxml2 on mw1017 (other canaries will be upgraded later on if all is well, the rest of mw* on Monday) |
[production] |
| 13:50 |
<apergos> |
on fluorine truncated archive/redis.log-20160311 and archive/JobQueueFederated.log-20160311 to 5mb each, they were each about 500gb |
[production] |
| 13:26 |
<mobrovac> |
restbase deploy end of 3bedb8f5c42 |
[production] |
| 12:56 |
<mobrovac> |
restbase deploy start of 3bedb8f5c42 on canary restbase1001 |
[production] |
| 12:36 |
<mobrovac> |
mathoid deploying 7a282a4181a4 |
[production] |
| 12:10 |
<moritzm> |
uploaded backport of linux-tools 4.4-4 for jessie-wikimedia to carbon (provides kbuild and perf amonst others) |
[production] |
| 11:48 |
<akosiaris> |
restarting gerrit on ytterbium |
[production] |
| 11:27 |
<jynus> |
running authdns-update to solve a dns typo |
[production] |
| 11:25 |
<jynus> |
killed some long-running git-upload-pack tasks |
[production] |
| 10:26 |
<hashar> |
Gerrit task management documented on https://wikitech.wikimedia.org/wiki/Gerrit#Tasks_management |
[production] |
| 10:11 |
<hashar> |
Gerrit: killed stuck "git-upload-pack '/mediawiki/core.git'" tasks |
[production] |
| 09:46 |
<elukey> |
Rebooting rdb200[56].codfw for kernel upgrade |
[production] |