|
2017-04-13
§
|
| 13:00 |
<moritzm> |
Upgrading thumbor* to Linux 4.9 |
[production] |
| 12:52 |
<elukey> |
temporary set config set client-output-buffer-limit "slave 5368709120 5368709120 180" on rdb1007:6379 |
[production] |
| 12:34 |
<volans@tin> |
Synchronized wmf-config/db-eqiad.php: Use a generic retry for the read only message T160178 (duration: 00m 44s) |
[production] |
| 12:34 |
<elukey> |
temporary set config set client-output-buffer-limit "slave 3221225472 3221225472 180" on rdb1007:6379 |
[production] |
| 12:22 |
<volans@tin> |
Synchronized wmf-config/db-codfw.php: Use a generic retry for the read only message T160178 (duration: 01m 54s) |
[production] |
| 12:16 |
<moritzm> |
restarting ntp on achernar |
[production] |
| 11:59 |
<elukey> |
temporary set config set client-output-buffer-limit "slave 2536870912 2536870912 60" on rdb1007:6379 |
[production] |
| 11:37 |
<elukey> |
temporary set config set client-output-buffer-limit "slave 2147483648 2147483648 60" on rdb1007:6379 to give time to rdb2005's replication to catch up - T159850 |
[production] |
| 10:58 |
<moritzm> |
rebooting alsafi to Linux 4.9 |
[production] |
| 10:58 |
<moritzm> |
rebooting alfafi to Linux 4.9 |
[production] |
| 10:47 |
<elukey> |
reverted previous config for Redis rdb2005 |
[production] |
| 10:47 |
<XioNoX> |
Confirmed we can still reach cr2-knams:lo0 via v6 (from esams), disabling IPv4 transit sessions for T162601 |
[production] |
| 10:42 |
<XioNoX> |
disable V6 transit BGP session on cr2-knams for T162601 |
[production] |
| 10:22 |
<elukey> |
executed CONFIG SET appendfsync no (prev value: "everysec") to Redis instance 6380 on rdb2005 - T125735 |
[production] |
| 10:13 |
<godog> |
upgrade thumbor to 0.1.38 |
[production] |
| 10:08 |
<moritzm> |
rebooting restbase1016 to Linux 4.9 |
[production] |
| 09:39 |
<moritzm> |
rebooting restbase1011 to Linux 4.9 |
[production] |
| 09:12 |
<moritzm> |
rebooting restbase1010 to Linux 4.9 |
[production] |
| 06:29 |
<elukey> |
re-arm keyholder on mira after reboot |
[production] |
| 06:14 |
<elukey> |
powercycle mira - eth0 errors in the dmesg, CPU system utilization skyrocketed |
[production] |
| 04:14 |
<mutante> |
ms-be2023 is rebooting |
[production] |
| 04:12 |
<mutante> |
ms-be2023 icinga alerts, no more swift processes. cant ssh to it. attempt to power cycle. mgmt console enourmous spam of "rejecting I/O to offline device" |
[production] |
| 01:58 |
<bblack@neodymium> |
conftool action : set/pooled=yes; selector: name=achernar.wikimedia.org,dc=codfw,cluster=dns,service=pdns_recursor |
[production] |
| 00:36 |
<catrope@tin> |
Finished scap: Split RCFilters GuidedTour messages for ORES vs non-ORES (T162693) (duration: 53m 47s) |
[production] |
|
2017-04-12
§
|
| 23:42 |
<catrope@tin> |
Started scap: Split RCFilters GuidedTour messages for ORES vs non-ORES (T162693) |
[production] |
| 23:37 |
<catrope@tin> |
Synchronized php-1.29.0-wmf.20/extensions/MobileFrontend/: Log only infoboxes which are not a direct children of lead section (T149884) (duration: 01m 05s) |
[production] |
| 23:35 |
<catrope@tin> |
Synchronized php-1.29.0-wmf.20/resources/src/mediawiki.widgets: Fix setDisabled in mw.widgets.Complex* (T162667) (duration: 00m 42s) |
[production] |
| 23:32 |
<catrope@tin> |
Synchronized php-1.29.0-wmf.19/resources/src/mediawiki.widgets: Fix setDisabled in mw.widgets.Complex* (T162667) (duration: 00m 44s) |
[production] |
| 23:25 |
<awight> |
rebuilt and reenabled process-control jobs |
[production] |
| 23:20 |
<catrope@tin> |
Synchronized wmf-config/InitialiseSettings.php: Disable cross-wiki uploads to Commons (T162374) (duration: 00m 43s) |
[production] |
| 23:19 |
<cwd> |
removed p-c crontab to stop all jobs |
[production] |
| 23:15 |
<bblack@neodymium> |
conftool action : set/pooled=no; selector: name=achernar.wikimedia.org,dc=codfw,cluster=dns,service=pdns_recursor |
[production] |
| 23:13 |
<catrope@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable wgCiteResponsiveReferences on cawiki (T161307) and bgwiki (T162145) (duration: 00m 44s) |
[production] |
| 23:02 |
<bblack@neodymium> |
conftool action : set/pooled=yes; selector: name=acamar.wikimedia.org,dc=codfw,cluster=dns,service=pdns_recursor |
[production] |
| 22:50 |
<bblack> |
acamar fixed up BIOS: HT disabled and power mgmt was set to PPW (DAPC) instead of PPW (OS) |
[production] |
| 22:45 |
<bblack> |
downtiming acamar again to fixup bios stuff (HT at least) |
[production] |
| 21:31 |
<Dereckson> |
Create Education Program tables on it.wikiversity (T162692) |
[production] |
| 20:44 |
<legoktm@tin> |
Synchronized wmf-config/InitialiseSettings.php: Deploy Linter to all wikis - T148609 (duration: 00m 44s) |
[production] |
| 20:42 |
<bblack@neodymium> |
conftool action : set/pooled=no; selector: name=acamar.wikimedia.org,dc=codfw,cluster=dns,service=pdns_recursor |
[production] |
| 20:25 |
<mutante> |
planet2001 - manually updating all feeds to make it active (or would have to wait for crons) |
[production] |
| 20:12 |
<ssastry@tin> |
Finished deploy [parsoid/deploy@323cebb]: Updating Parsoid to 75debae3 (duration: 09m 16s) |
[production] |
| 20:07 |
<mutante> |
planet2001 - activating all the crons, making planet active/active eqiad/codfw |
[production] |
| 20:03 |
<ssastry@tin> |
Started deploy [parsoid/deploy@323cebb]: Updating Parsoid to 75debae3 |
[production] |
| 19:42 |
<bd808@tin> |
Synchronized wmf-config/mc.php: Revert "wikitech: Enable binary memcached protocol" (duration: 00m 43s) |
[production] |
| 19:36 |
<bd808> |
testing stashbot post-restart |
[production] |
| 19:33 |
<demon@tin> |
Synchronized wmf-config/mc.php: Bryan made me do it (duration: 00m 43s) |
[production] |
| 19:05 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.20 |
[production] |
| 19:05 |
<XenoRyet> |
reverted SmashPig from aede2772341b321aacbd970ec2b68655b704b147 to ab52dbe6628af2ebc717603259b584031807f6fe |
[production] |
| 19:05 |
<demon@tin> |
Synchronized php: symlink bump (duration: 00m 42s) |
[production] |
| 19:04 |
<ejegg> |
updated payments-wiki from 0b396a397272ed36325fcd944a3d0ba3625c212c to 36f38f6c96887f328596a1b01ad6a03ef75a1b29 |
[production] |