2901-2950 of 10000 results (39ms)
2016-04-21 §
02:22 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.21) (duration: 09m 48s) [production]
01:49 <mutante> git pull on strontium, ops/puppet [production]
01:48 <mutante> belated log: restarted slapd on seaborgium [production]
01:29 <ori> installed python-progressbar on terbium for warmup script, will be puppetized later [production]
2016-04-20 §
22:18 <mutante> creating ganeti VM install1001 on eqiad cluster [production]
19:03 <AaronSchulz> Cleared out 'enqueue' job queues to see if corruption comes back [production]
18:17 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Promote db1031 as the new x1 eqiad local master (duration: 00m 28s) [production]
18:16 <ori@tin> Synchronized php-1.27.0-wmf.21/extensions/Translate/messagegroups/WikiPageMessageGroup.php: I331bd93b: Avoid more master queries on page views (duration: 00m 31s) [production]
18:16 <ori@tin> Synchronized php-1.27.0-wmf.21/includes/jobqueue/JobQueueGroup.php: Ie9799f5ea: Catch errors in pushLazyJobs() and log them (duration: 00m 36s) [production]
17:59 <jynus> changing database topology to set db1031 as the master of x1 on eqiad [production]
17:58 <volans> Upgrading db1065 and fixing overheathing problems T132515 [production]
17:30 <volans> Upgrading db1070 and fixing overheathing problems T132515 [production]
17:19 <aaron@tin> Synchronized php-1.27.0-wmf.21/includes/jobqueue/JobQueueRedis.php: 86d185a4bbf52d (duration: 00m 39s) [production]
17:15 <volans> Upgrading db1071 and fixing overheathing problems T132515 [production]
17:03 <akosiaris> aptitude purge php5-xhprof on uranium [production]
16:54 <elukey> replaced "#" with ";" manually in uranium's /etc/php5/cli/conf.d/20-xhprof.ini and /etc/php5/apache2/php.ini to avoid cronspam (didn't find puppet/package trails) [production]
15:43 <ebernhardson> delete apifeatureusage-2016.01.20 from codfw elasticsearch cluster. Index should never have existed in this cluster (and is beyond retention). [production]
15:42 <ebernhardson> delete apifeatureusage-2016-01-(02,09,10) from eqiad elasticsearch cluster. We only keep 30 days of apifeatureusage logs [production]
15:37 <jynus@tin> Synchronized wmf-config/db-codfw.php: Tweak DB weights for better latency, avoiding peaks on QPS (duration: 00m 32s) [production]
15:18 <ottomata> enabling puppet on analytics1015 [production]
15:17 <andrewbogott> re-imaging labtestvirt2001 and labtestneutron2001 [production]
14:56 <volans@tin> Synchronized wmf-config/db-eqiad.php: Change eqiad masters for s1,s3-s7 - T105135 (duration: 00m 28s) [production]
14:55 <ottomata> started puppet on analytics1003 [production]
14:52 <jynus@tin> Synchronized wmf-config/db-codfw.php: Repool es2019 (duration: 00m 38s) [production]
14:37 <ottomata> stopping puppet on analytics1015 and analytics1003 in prep for migration [production]
13:54 <elukey> puppet disabled on analytics1027 to stop Camus [production]
13:50 <_joe_> rolling restart of ocg servers [production]
13:21 <moritzm> rebooting rdb1002,rdb1003,rdb1004,rdb1006,rdb1007,rdb1008 for upgrade to Linux 4.4 [production]
13:17 <jynus> [switchover-maintenance] Changing DB slave topology for shard s1 on eqiad T111654 [production]
12:59 <volans> [switchover-maintenance] Changing DB slave topology for shard s4 on eqiad T111654 [production]
12:54 <volans> [switchover-maintenance] Changing DB slave topology for shard s5 on eqiad T111654 [production]
12:48 <jynus> [switchover-maintenance] Changing DB slave topology for shard s3 on eqiad T111654 [production]
12:37 <volans> [switchover-maintenance] Changing DB slave topology for shard s6 on eqiad T111654 [production]
12:17 <volans> [switchover-maintenance] Changing DB slave topology for shard s7 on eqiad T111654 [production]
10:51 <godog> pool restbase1014 [production]
10:42 <volans> [switchover-maintenance] Restarting db1028 (s7) [production]
10:33 <jynus> [switchover-maintenance] Restarting db1018 [production]
10:31 <volans> [switchover-maintenance] Upgrading TLS for shard s7 on eqiad databases [production]
09:13 <jynus> backfilling recentchanges on enwiki API servers [production]
09:12 <godog> stop compactions on restbase1014-[ab] [production]
08:54 <elukey> deployed the new puppet compiler - version 0.1.4 (hosts sorted in the HTML output, minor change) [production]
02:31 <l10nupdate@tin> ResourceLoader cache refresh completed at Wed Apr 20 02:31:20 UTC 2016 (duration 8m 46s) [production]
02:22 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.21) (duration: 09m 37s) [production]
2016-04-19 §
22:42 <ori> killing rc insert query on db1065 and db1066 [production]
21:27 <ori> running rebuildrecentchanges.php --from=20160419144741 --to=20160419151018 on all wikis [production]
21:12 <paravoid> clearing the exim4 retry database on mx2001 [production]
20:44 <ori> on all wikis, deleting from recentchanges where rc_timestamp > 20160419144741 and rc_timestamp < 20160419151018 [production]
20:09 <ori> ran `mwscript rebuildrecentchanges.php --wiki=testwiki --from=20160419144741 --to=20160419151018` [production]
19:53 <paravoid> staggered varnish bans for 'obj.http.server ~ "^mw2.+"' as a workaround for T133069 [production]
19:51 <ori@tin> Synchronized php-1.27.0-wmf.21/maintenance/rebuildrecentchanges.php: Ie9799f5ea: rebuildrecentchanges: Allow rebuilding specified time range only (duration: 00m 28s) [production]