|
2018-10-25
ยง
|
| 21:35 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: rolling back group1 refs T206655 T208000 |
[production] |
| 21:29 |
<XioNoX> |
configure 208.80.153.185/29 on cr1/2-codfw - T207663 |
[production] |
| 21:25 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: group2 wikis to 1.33.0-wmf.1 refs T206655 |
[production] |
| 20:48 |
<twentyafterfour> |
staying at group1, error rate seems to have stabilized |
[production] |
| 20:43 |
<ejegg> |
updated fundraising python tools from 5a2d39b41b to af5dbee8eb |
[production] |
| 20:37 |
<ejegg> |
updated standalone SmashPig deployment from b638ca02bc to f65daa8550 |
[production] |
| 20:32 |
<twentyafterfour> |
db error rate increased again. rolling back |
[production] |
| 20:31 |
<twentyafterfour@deploy1001> |
Synchronized php: group1 wikis to 1.33.0-wmf.1 refs T206655 (duration: 00m 54s) |
[production] |
| 20:30 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.1 refs T206655 |
[production] |
| 20:07 |
<jgleeson_> |
Updated paymentswiki from 5a7a8e7e4f to f5999d963d |
[production] |
| 20:04 |
<twentyafterfour> |
still haven't deployed wmf.1 yet error rate increased and icinga is alerting about mediawiki exceptions + wdqs1010 degraded |
[production] |
| 20:02 |
<twentyafterfour@deploy1001> |
Finished scap: full sync to be sure that 1.33.0-wmf.1 is fully deployed (duration: 36m 57s) |
[production] |
| 19:54 |
<mutante> |
mw1272 - repooled (T207983) |
[production] |
| 19:51 |
<mutante> |
mw1272 - rebooting (a stop job is running for HHVM PH/Hack runtime) (T207983) |
[production] |
| 19:47 |
<mutante> |
mw1272 - depooled, restarting hhvm (T207983) |
[production] |
| 19:45 |
<mutante> |
mw1272 - depooled |
[production] |
| 19:26 |
<twentyafterfour@deploy1001> |
Started scap: full sync to be sure that 1.33.0-wmf.1 is fully deployed |
[production] |
| 19:23 |
<twentyafterfour> |
beginning mediawiki train. Will start with group1 and then monitor the situation for a few minutes. If everything looks good then we go to group2. |
[production] |
| 19:20 |
<sbisson@deploy1001> |
Synchronized php-1.32.0-wmf.26/extensions/ContentTranslation/: SWAT: [[gerrit:469603|Add detailed logging for AbuseFilter]] (duration: 00m 56s) |
[production] |
| 18:37 |
<sbisson@deploy1001> |
Synchronized php-1.33.0-wmf.1/extensions/ContentTranslation/: SWAT: [[gerrit:469605|Remove the session parameter from AbuseFilter logging]] (duration: 00m 56s) |
[production] |
| 17:58 |
<aaron@deploy1001> |
Synchronized php-1.33.0-wmf.1/extensions/Translate/tag: c5fa239917a870240ec4dcd8a617f0f8033aa9bf (duration: 00m 55s) |
[production] |
| 17:52 |
<aaron@deploy1001> |
Synchronized php-1.33.0-wmf.1/includes/page/WikiPage.php: f3b5a1df116f426c2809f2a266b9d761f15c349f (duration: 00m 54s) |
[production] |
| 17:52 |
<smalyshev@deploy1001> |
Finished deploy [wdqs/wdqs@4967dba]: Test deploy new update & scripts (duration: 00m 28s) |
[production] |
| 17:51 |
<smalyshev@deploy1001> |
Started deploy [wdqs/wdqs@4967dba]: Test deploy new update & scripts |
[production] |
| 17:50 |
<aaron@deploy1001> |
Synchronized php-1.33.0-wmf.1/tests/phpunit/includes/page/WikiPageDbTestBase.php: f3b5a1df116f426c2809f2a266b9d761f15c349f (duration: 00m 55s) |
[production] |
| 17:36 |
<aaron@deploy1001> |
Synchronized php-1.33.0-wmf.1/includes/changetags/ChangeTags.php: 08f8e6a9d7f1dcb281321c5e3a3471169e68348d (duration: 00m 55s) |
[production] |
| 17:24 |
<bsitzmann@deploy1001> |
Finished deploy [mobileapps/deploy@95452cf]: Update mobileapps to 58cbdff (T206527) (duration: 03m 50s) |
[production] |
| 17:20 |
<bsitzmann@deploy1001> |
Started deploy [mobileapps/deploy@95452cf]: Update mobileapps to 58cbdff (T206527) |
[production] |
| 17:20 |
<mutante> |
planet - regenerating feeds for 'en' and 'de', others will follow by cron. switching to new theme. replaced bootstrap with bulma. removed jQuery. thanks to paladox |
[production] |
| 16:34 |
<gehel@puppetmaster1001> |
conftool action : set/weight=20; selector: dc=eqiad,cluster=wdqs,name=wdqs1005.codfw.wmnet |
[production] |
| 16:34 |
<gehel@puppetmaster1001> |
conftool action : set/weight=20; selector: dc=eqiad,cluster=wdqs,name=wdqs1004.codfw.wmnet |
[production] |
| 16:34 |
<gehel> |
decreasing relative weight of wdqs1003 in LVS to ease the updater |
[production] |
| 16:24 |
<shdubsh> |
installed patched nagios-nrpe-plugin and nagios-nrpe-server on icinga1001 - T207775 |
[production] |
| 15:36 |
<elukey> |
shutdown aqs1006 to replace one broken disk - T206915 |
[production] |
| 15:31 |
<SMalyshev> |
depooling wdqs1003 again, it's not catching up like the other hosts |
[production] |
| 15:16 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: [Beta Cluster] Re-enable WBMI on Beta Commons (duration: 00m 54s) |
[production] |
| 15:11 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Revert "logging: Disable Wikibase.NewItemIdFormatter channel" (duration: 00m 55s) |
[production] |
| 15:08 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Explicitly set wgLexemeEnableRepo for wikidatas [[gerrit:469625]] (duration: 00m 55s) |
[production] |
| 15:02 |
<godog> |
test rsyslog 8.38 upgrade on lithium - T136312 |
[production] |
| 14:28 |
<elukey> |
upgrade druid on druid100[4-6] to Druid 0.12.3 |
[production] |
| 14:20 |
<banyek> |
running dns update (gerrit patch: 467711) |
[production] |
| 13:48 |
<anomie@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Setting comment table migration stage to write-new/read-both on all wikis (T166733) (duration: 00m 55s) |
[production] |
| 13:46 |
<godog> |
reformat ms-be2043 xfs filesystems - T199198 |
[production] |
| 13:29 |
<XioNoX> |
test successful, rollback add term return-tcp permit on cr2-codfw |
[production] |
| 13:28 |
<XioNoX> |
test add term return-tcp permit on cr2-codfw |
[production] |
| 12:14 |
<volans> |
rebooting cumin1001 to pick new kernel and clear any potential weird state after OOMs |
[production] |
| 12:01 |
<zeljkof> |
EU SWAT finished |
[production] |
| 11:17 |
<zfilipin@deploy1001> |
Synchronized wmf-config/throttle.php: SWAT: [[gerrit:469261|New throttle rule for Johannesburg Event on 2018-10-27 (T207742)]] (duration: 00m 55s) |
[production] |
| 11:08 |
<zfilipin@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:465418|Stop collecting data CitaitonUsage and CitationUsagePageLoad (T191086 T203253)]] (duration: 00m 57s) |
[production] |
| 10:57 |
<volans> |
restart pdfrender on scb1003 |
[production] |