1-50 of 10000 results (47ms)
2018-12-30 §
07:17 <elukey> restart pdfrender on scb1002 (alarms flapping) [production]
02:07 <bawolff@deploy1001> Synchronized private/PrivateSettings.php: fine-tune antispam measure T212667 (duration: 00m 47s) [production]
2018-12-29 §
14:53 <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T212667 218371fd35 - Adjust mw.org abusefilter emergency shutoff threshold down to 0.3 (duration: 00m 46s) [production]
14:30 <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T212667 fe72284c Adjust account throttle limits (duration: 00m 46s) [production]
13:30 <bawolff@deploy1001> Synchronized private/PrivateSettings.php: T212667 - adjust spam block (duration: 00m 44s) [production]
12:34 <bawolff@deploy1001> Synchronized private/PrivateSettings.php: T212667 - make spam mitigation global (duration: 00m 49s) [production]
11:52 <volans|off> manually recreated /dev/log symlink on kubernetes1001, restarting systemd-journald.socket didn't worked (this should fix cron-spam emails from the host every hour) [production]
11:43 <volans|off> restarted pdfrender on scb1004 [production]
09:21 <elukey> restart pdfrender on scb1004 [production]
2018-12-28 §
18:14 <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 97446843a27 T212667 - Temp increase abusefilter emergency cutoff on mw.org to deal with spam attack (duration: 00m 46s) [production]
18:02 <bawolff@deploy1001> Synchronized private/PrivateSettings.php: T212667 - adjust account creation (duration: 00m 47s) [production]
15:28 <bawolff@deploy1001> Synchronized private/PrivateSettings.php: Attempt to adjust captcha settings for T212667 (duration: 00m 46s) [production]
2018-12-27 §
19:31 <mobrovac@deploy1001> Finished deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays), take #3 (duration: 25m 49s) [production]
19:05 <mobrovac@deploy1001> Started deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays), take #3 [production]
18:15 <mobrovac@deploy1001> Started deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays), take #2 [production]
18:15 <mobrovac@deploy1001> Finished deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays) - T212631 (duration: 11m 20s) [production]
18:04 <mobrovac@deploy1001> Started deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays) - T212631 [production]
18:04 <mobrovac@deploy1001> deploy aborted: Fix rate-limiter crash (with increased deploy delays) - T212631 (duration: 00m 09s) [production]
18:04 <mobrovac@deploy1001> Started deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays) - T212631 [production]
18:03 <mobrovac@deploy1001> deploy aborted: Fix rate-limiter crash - T212631 (duration: 13m 09s) [production]
17:50 <mobrovac@deploy1001> Started deploy [restbase/deploy@70c4752]: Fix rate-limiter crash - T212631 [production]
17:20 <mobrovac@deploy1001> Finished deploy [restbase/deploy@ae7a537]: Fix rate-limiter crash - T212631 - deploy only on canary restbase1007 (duration: 04m 24s) [production]
17:15 <mobrovac@deploy1001> Started deploy [restbase/deploy@ae7a537]: Fix rate-limiter crash - T212631 - deploy only on canary restbase1007 [production]
16:00 <cmjohnson1> powercycling frdb1001 for troubleshooting [production]
15:21 <bawolff> Updated interwiki cache https://biblio.wiki to https://wikilivres.org -> T212650 [production]
15:19 <bawolff@deploy1001> Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 03m 34s) [production]
2018-12-26 §
16:37 <godog> clear logstash persistent queue /var/lib/logstash on logstash100[789] [production]
15:47 <godog> roll-restart logstash on logstash100[789] [production]
14:26 <filippo@puppetmaster1001> conftool action : set/pooled=no; selector: name=wtp1028.eqiad.wmnet [production]
14:18 <godog> powercycle wtp1028 - nothing on console [production]
09:13 <godog> swift eqiad-prod: more weight for ms-be10[44-50].eqiad.wmnet - T209618 [production]
07:20 <marostegui> Fix replication db1124:3318 [production]
2018-12-24 §
05:38 <_joe_> powercycling lvs3001 [production]
2018-12-23 §
11:53 <marostegui> Fix replication on db1124:3318 [production]
10:25 <ariel@deploy1001> Finished deploy [dumps/dumps@af74350]: python3 fixup for show runtimes (duration: 00m 05s) [production]
10:25 <ariel@deploy1001> Started deploy [dumps/dumps@af74350]: python3 fixup for show runtimes [production]
08:53 <apergos> restarted pdfrender on scb1003 [production]
2018-12-22 §
18:45 <elukey> manually clean up of old log files on an-coord1001 (disk space issues) [production]
15:58 <godog> reboot ms-be2018, stuck on sd 0:1:0:1: rejecting I/O to offline device [production]
10:21 <cwd> re-enabled process-control [production]
08:36 <cwd> took down sidebar fr campaign [production]
08:36 <cwd> disabled process-control [production]
01:00 <AaronSchulz> Deployed b47e9fcfece99 to navtiming [production]
00:59 <aaron@deploy1001> Finished deploy [performance/navtiming@b47e9fc]: (no justification provided) (duration: 00m 05s) [production]
00:59 <aaron@deploy1001> Started deploy [performance/navtiming@b47e9fc]: (no justification provided) [production]
2018-12-21 §
21:29 <mutante> phab1002 - temp hack to unbreak phd / systemd alert, real fix will be phab deployment to new server [production]
21:28 <mutante> phab1002 - mkdir -p /srv/phab/libext/ava/src ; touch __phutil_library_init__.php [production]
21:05 <mutante> phab1002 - apt autoremove [production]
21:04 <mutante> phab1002 - removing all php related packages and letting puppet reinstall them [production]
20:38 <mutante> phab1002 - restart php-fpm, restart phd for testing. phd fails [production]