1551-1600 of 10000 results (38ms)
2017-04-13 §
16:55 <elukey> restored default value of client-output-buffer-limit on rdb1007:6379 - T159850 [production]
16:23 <mobrovac@tin> Finished deploy [citoid/deploy@b8c4cb2]: Test deploy for T162814 (duration: 02m 24s) [production]
16:21 <mobrovac@tin> Started deploy [citoid/deploy@b8c4cb2]: Test deploy for T162814 [production]
16:15 <thcipriani@tin> Synchronized README: [[gerrit:347918|scap.cfg change test]] (duration: 00m 44s) [production]
15:49 <mobrovac@tin> Finished deploy [citoid/deploy@212800d]: Enable multiple results for T115248 and remove b/c for T114515 (duration: 03m 10s) [production]
15:46 <mobrovac@tin> Started deploy [citoid/deploy@212800d]: Enable multiple results for T115248 and remove b/c for T114515 [production]
15:02 <andrewbogott> disabling puppet on dubnium and pollux for a cautious merge of https://gerrit.wikimedia.org/r/#/c/348071 [production]
15:01 <andrewbogott> disabling puppet on seaborgium and serpens for a cautious merge of https://gerrit.wikimedia.org/r/#/c/348071 [production]
14:56 <ppchelko@tin> Finished deploy [changeprop/deploy@e47afea]: Provide separate rules for ORES precaching in both DCs (duration: 00m 58s) [production]
14:55 <ppchelko@tin> Started deploy [changeprop/deploy@e47afea]: Provide separate rules for ORES precaching in both DCs [production]
14:50 <moritzm> installing bouncycastle security updates [production]
14:27 <bblack> disabling puppet on recnds/ntp boxes to control patch rollout [production]
13:28 <moritzm> powercycling thumbor1001, stuck in reboot [production]
13:18 <hashar@tin> Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 43s) [production]
13:16 <hashar@tin> Synchronized dblists/closed.dblist: Close wikimania2016 - T161183 (duration: 00m 43s) [production]
13:14 <hashar@tin> Synchronized static/images/project-logos: (no justification provided) (duration: 00m 46s) [production]
13:00 <moritzm> Upgrading thumbor* to Linux 4.9 [production]
12:52 <elukey> temporary set config set client-output-buffer-limit "slave 5368709120 5368709120 180" on rdb1007:6379 [production]
12:34 <volans@tin> Synchronized wmf-config/db-eqiad.php: Use a generic retry for the read only message T160178 (duration: 00m 44s) [production]
12:34 <elukey> temporary set config set client-output-buffer-limit "slave 3221225472 3221225472 180" on rdb1007:6379 [production]
12:22 <volans@tin> Synchronized wmf-config/db-codfw.php: Use a generic retry for the read only message T160178 (duration: 01m 54s) [production]
12:16 <moritzm> restarting ntp on achernar [production]
11:59 <elukey> temporary set config set client-output-buffer-limit "slave 2536870912 2536870912 60" on rdb1007:6379 [production]
11:37 <elukey> temporary set config set client-output-buffer-limit "slave 2147483648 2147483648 60" on rdb1007:6379 to give time to rdb2005's replication to catch up - T159850 [production]
10:58 <moritzm> rebooting alsafi to Linux 4.9 [production]
10:58 <moritzm> rebooting alfafi to Linux 4.9 [production]
10:47 <elukey> reverted previous config for Redis rdb2005 [production]
10:47 <XioNoX> Confirmed we can still reach cr2-knams:lo0 via v6 (from esams), disabling IPv4 transit sessions for T162601 [production]
10:42 <XioNoX> disable V6 transit BGP session on cr2-knams for T162601 [production]
10:22 <elukey> executed CONFIG SET appendfsync no (prev value: "everysec") to Redis instance 6380 on rdb2005 - T125735 [production]
10:13 <godog> upgrade thumbor to 0.1.38 [production]
10:08 <moritzm> rebooting restbase1016 to Linux 4.9 [production]
09:39 <moritzm> rebooting restbase1011 to Linux 4.9 [production]
09:12 <moritzm> rebooting restbase1010 to Linux 4.9 [production]
06:29 <elukey> re-arm keyholder on mira after reboot [production]
06:14 <elukey> powercycle mira - eth0 errors in the dmesg, CPU system utilization skyrocketed [production]
04:14 <mutante> ms-be2023 is rebooting [production]
04:12 <mutante> ms-be2023 icinga alerts, no more swift processes. cant ssh to it. attempt to power cycle. mgmt console enourmous spam of "rejecting I/O to offline device" [production]
01:58 <bblack@neodymium> conftool action : set/pooled=yes; selector: name=achernar.wikimedia.org,dc=codfw,cluster=dns,service=pdns_recursor [production]
00:36 <catrope@tin> Finished scap: Split RCFilters GuidedTour messages for ORES vs non-ORES (T162693) (duration: 53m 47s) [production]
2017-04-12 §
23:42 <catrope@tin> Started scap: Split RCFilters GuidedTour messages for ORES vs non-ORES (T162693) [production]
23:37 <catrope@tin> Synchronized php-1.29.0-wmf.20/extensions/MobileFrontend/: Log only infoboxes which are not a direct children of lead section (T149884) (duration: 01m 05s) [production]
23:35 <catrope@tin> Synchronized php-1.29.0-wmf.20/resources/src/mediawiki.widgets: Fix setDisabled in mw.widgets.Complex* (T162667) (duration: 00m 42s) [production]
23:32 <catrope@tin> Synchronized php-1.29.0-wmf.19/resources/src/mediawiki.widgets: Fix setDisabled in mw.widgets.Complex* (T162667) (duration: 00m 44s) [production]
23:25 <awight> rebuilt and reenabled process-control jobs [production]
23:20 <catrope@tin> Synchronized wmf-config/InitialiseSettings.php: Disable cross-wiki uploads to Commons (T162374) (duration: 00m 43s) [production]
23:19 <cwd> removed p-c crontab to stop all jobs [production]
23:15 <bblack@neodymium> conftool action : set/pooled=no; selector: name=achernar.wikimedia.org,dc=codfw,cluster=dns,service=pdns_recursor [production]
23:13 <catrope@tin> Synchronized wmf-config/InitialiseSettings.php: Enable wgCiteResponsiveReferences on cawiki (T161307) and bgwiki (T162145) (duration: 00m 44s) [production]
23:02 <bblack@neodymium> conftool action : set/pooled=yes; selector: name=acamar.wikimedia.org,dc=codfw,cluster=dns,service=pdns_recursor [production]