5601-5650 of 10000 results (54ms)
2019-01-02 §
11:17 <ema@puppetmaster1001> conftool action : set/pooled=yes; selector: name=ms-fe2006.codfw.wmnet [production]
11:10 <ema@puppetmaster1001> conftool action : set/pooled=no; selector: name=ms-fe2006.codfw.wmnet [production]
10:59 <ema> replace TLS certificates on ms-fe codfw hosts T212215 [production]
10:52 <moritzm> rebooting centrallog1001 for kernel security update [production]
10:48 <volans> testing the new spicerack package on cumin2001, in the unlikely event you need to use spicerack cookbooks today please use cumin1001 [production]
10:45 <godog> ms-be2018 Flashing Smart Array P840 in Slot 3 [ 3.00 -> 6.60 ] [production]
10:43 <moritzm> removed labvirt1013 from debmonitor, got renamed in T212513 [production]
10:42 <volans> uploaded spicerack_0.0.10-1_amd64.deb to apt.wikimedia.org stretch-wikimedia [production]
10:03 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Repool db2096 (duration: 00m 44s) [production]
09:50 <marostegui> Stop MySQL on db2096 for kernel and mysql upgrade [production]
09:49 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Depool db2096 (duration: 00m 45s) [production]
09:48 <marostegui@deploy1001> sync-file aborted: Depool db2096 (duration: 00m 01s) [production]
09:18 <moritzm> installing c3p0 security updates [production]
09:07 <Zoranzoki21> Drop valid_tag from s8 by Marostegui - T212254 [production]
09:06 <godog> eqiad-prod: final weight for ms-be10[44-50].eqiad.wmnet - T209618 [production]
08:56 <moritzm> installing libarchive security updates [production]
07:38 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1078 - T212692 (duration: 00m 46s) [production]
07:30 <marostegui> Fix login.logging table on db1078 - T212692 [production]
07:30 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1078 - T212692 (duration: 00m 47s) [production]
07:00 <marostegui> Deploy schema change on s1 codfw master (lag will be generated on s1 codfw) - T202167 T86338 [production]
06:54 <marostegui> Drop empty valid_tag table from labswiki labtestwiki - T212254 [production]
06:49 <marostegui> Drop empty valid_tag table from s5 - T212254 [production]
06:25 <marostegui> Drop valid_tag from s6 - T212254 [production]
06:15 <marostegui> Fix last chunks on db1124:338 - T212574 [production]
2018-12-31 §
16:27 <ejegg> disabled SmashPig recurring donation charge job for investigation [production]
2018-12-30 §
07:17 <elukey> restart pdfrender on scb1002 (alarms flapping) [production]
02:07 <bawolff@deploy1001> Synchronized private/PrivateSettings.php: fine-tune antispam measure T212667 (duration: 00m 47s) [production]
2018-12-29 §
14:53 <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T212667 218371fd35 - Adjust mw.org abusefilter emergency shutoff threshold down to 0.3 (duration: 00m 46s) [production]
14:30 <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T212667 fe72284c Adjust account throttle limits (duration: 00m 46s) [production]
13:30 <bawolff@deploy1001> Synchronized private/PrivateSettings.php: T212667 - adjust spam block (duration: 00m 44s) [production]
12:34 <bawolff@deploy1001> Synchronized private/PrivateSettings.php: T212667 - make spam mitigation global (duration: 00m 49s) [production]
11:52 <volans|off> manually recreated /dev/log symlink on kubernetes1001, restarting systemd-journald.socket didn't worked (this should fix cron-spam emails from the host every hour) [production]
11:43 <volans|off> restarted pdfrender on scb1004 [production]
09:21 <elukey> restart pdfrender on scb1004 [production]
2018-12-28 §
18:14 <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 97446843a27 T212667 - Temp increase abusefilter emergency cutoff on mw.org to deal with spam attack (duration: 00m 46s) [production]
18:02 <bawolff@deploy1001> Synchronized private/PrivateSettings.php: T212667 - adjust account creation (duration: 00m 47s) [production]
15:28 <bawolff@deploy1001> Synchronized private/PrivateSettings.php: Attempt to adjust captcha settings for T212667 (duration: 00m 46s) [production]
2018-12-27 §
19:31 <mobrovac@deploy1001> Finished deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays), take #3 (duration: 25m 49s) [production]
19:05 <mobrovac@deploy1001> Started deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays), take #3 [production]
18:15 <mobrovac@deploy1001> Started deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays), take #2 [production]
18:15 <mobrovac@deploy1001> Finished deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays) - T212631 (duration: 11m 20s) [production]
18:04 <mobrovac@deploy1001> Started deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays) - T212631 [production]
18:04 <mobrovac@deploy1001> deploy aborted: Fix rate-limiter crash (with increased deploy delays) - T212631 (duration: 00m 09s) [production]
18:04 <mobrovac@deploy1001> Started deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays) - T212631 [production]
18:03 <mobrovac@deploy1001> deploy aborted: Fix rate-limiter crash - T212631 (duration: 13m 09s) [production]
17:50 <mobrovac@deploy1001> Started deploy [restbase/deploy@70c4752]: Fix rate-limiter crash - T212631 [production]
17:20 <mobrovac@deploy1001> Finished deploy [restbase/deploy@ae7a537]: Fix rate-limiter crash - T212631 - deploy only on canary restbase1007 (duration: 04m 24s) [production]
17:15 <mobrovac@deploy1001> Started deploy [restbase/deploy@ae7a537]: Fix rate-limiter crash - T212631 - deploy only on canary restbase1007 [production]
16:00 <cmjohnson1> powercycling frdb1001 for troubleshooting [production]
15:21 <bawolff> Updated interwiki cache https://biblio.wiki to https://wikilivres.org -> T212650 [production]