4201-4250 of 10000 results (42ms)
2015-09-21 §
04:31 <l10nupdate@tin> ResourceLoader cache refresh completed at Mon Sep 21 04:31:03 UTC 2015 (duration 31m 2s) [production]
02:23 <l10nupdate@tin> LocalisationUpdate completed (1.26wmf23) at 2015-09-21 02:23:12+00:00 [production]
02:19 <l10nupdate@tin> Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 25s) [production]
02:06 <MaxSem> Maps: created indexes on admin. <3 Postgres :( [production]
01:56 <bblack> downtimed eqiad ipv6 text/upload alerts as well, as with mobile above ( 1 301 TLS Redirect - 505 bytes in 1.008 second response time [production]
01:46 <bblack> downtimed the "LVS HTTP IPv6 on mobile-lb.eqiad.wikimedia.org_ipv6" alert for now ( https://phabricator.wikimedia.org/T113154 ) [production]
2015-09-20 §
22:34 <yuvipanda> reloda pybal on lvs1012 [production]
17:01 <bblack> repooling cp1046 varnish-be + varnish-be-rand in confctl, fresh storage, purge queue caught up - T113184 [production]
16:44 <bblack> depooling cp1046 varnish-be + varnish-be-rand in confctl, wiping storage, re-pooling - T113184 [production]
07:24 <paravoid> temporarily disabling puppet on fermium and applying antispam countermeasures [production]
04:29 <l10nupdate@tin> ResourceLoader cache refresh completed at Sun Sep 20 04:29:16 UTC 2015 (duration 29m 15s) [production]
02:22 <l10nupdate@tin> LocalisationUpdate completed (1.26wmf23) at 2015-09-20 02:22:53+00:00 [production]
02:19 <l10nupdate@tin> Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 12s) [production]
2015-09-19 §
23:12 <urandom> begining Cassandra repair on restbase1005 (nodetool repair -pr) [production]
23:08 <urandom> begining Cassandra repair on restbase1004 (nodetool repair -pr) [production]
19:56 <jynus> restarting once more giblit, last chance [production]
19:04 <paravoid> salt rm /etc/systemd/system/txstatsd.service from all cp*, leftover because of ::txstatsd::decommission (removed with 4a1d4e) missing it [production]
18:45 <_joe_> restarted gitblit. I will now substitute myself with a clever perl one-liner. [production]
18:38 <paravoid> pooling back cp1046 to pybal eqiad/mobile, has stayed stable [production]
18:34 <paravoid> reactivating ΒGP with GTT @ eqiad [production]
08:42 <_joe_> cp1046 dead on console again, powercycling to inspect it [production]
05:49 <aaron@tin> Synchronized php-1.26wmf23/extensions/TitleBlacklist: 80d3a21a51f9c54ed2d94 (duration: 00m 12s) [production]
05:22 <paravoid> pybal-depooling cp1046 from eqiad/mobile until further investigation [production]
05:21 <paravoid> powercycling cp1046, dead on console [production]
04:28 <l10nupdate@tin> ResourceLoader cache refresh completed at Sat Sep 19 04:28:59 UTC 2015 (duration 28m 57s) [production]
02:23 <l10nupdate@tin> LocalisationUpdate completed (1.26wmf23) at 2015-09-19 02:23:29+00:00 [production]
02:20 <l10nupdate@tin> Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 05s) [production]
2015-09-18 §
20:52 <ebernhardson> restart es on elastic1025 to disable dynamic scripting [production]
20:34 <gwicke> dropped by_ns indexes on restbase title_revisions tables [production]
19:54 <gwicke> finished deploy of restbase daacf4daa [production]
19:45 <gwicke> re-enabled puppet on restbase100* [production]
19:35 <gwicke> canary deploy of restbase daacf4daa on restbase1001; moving forward so that we can re-enable puppet over the weekend. [production]
17:54 <ebernhardson@tin> Synchronized wmf-config/CommonSettings.php: Replace insecure es usage with usage of a plugin (duration: 00m 12s) [production]
16:41 <mutante> mailman now on 2.1.18 and jessie [production]
16:14 <dcausse> elastic in eqiad plugin updates: restarting elastic1021 [production]
16:07 <paravoid> deactivating ΒGP with GTT @ eqiad [production]
15:20 <godog> create restbase user on cassandra test cluster [production]
14:55 <dcausse> elastic in eqiad plugin updates: restarting elastic1020 [production]
14:22 <bblack> committing lvs1007-1012 port/vlan changes for asw-d-eqiad (but leaving all 6 LVS ports in "disabled" state - T112781 ) [production]
14:14 <bblack> committing lvs1007-12 port/vlan changes for asw-b-eqiad, round 3... [production]
14:11 <mutante> sodium - stopped exim - rsyncing lists to fermium [production]
14:10 <dcausse> elastic in eqiad plugin updates: restarting elastic1019 [production]
14:07 <mutante> stopped mailman on sodium [production]
14:01 <bblack> rollback on asw-b-eqiad changes above [production]
13:56 <bblack> committing eqiad lvs1007-1012 port/vlan changes for asw-b-eqiad [production]
13:20 <bblack> committing eqiad lvs1007-12 port/vlan changes for asw-c-eqiad [production]
13:16 <bblack> commiting eqiad lvs1007-12 port/vlan changes for asw2-a5-eqiad [production]
13:12 <dcausse> elastic in eqiad plugin updates: restarting elastic1018 [production]
12:21 <godog> restart logstash on logstash1001, OOM in logs [production]
11:55 <dcausse> elastic in eqiad plugin updates: restarting elastic1017 [production]