|
2015-09-21
§
|
| 08:29 |
<moritzm> |
repooled mw1190-mw1195 and mw1197-mw1199 (for T104968) |
[production] |
| 08:21 |
<_joe_> |
restarted the logstash agent on logstash1003, OOM'd |
[production] |
| 08:18 |
<moritzm> |
depooled mw1190-mw1195 and mw1197-mw1199 (for T104968) |
[production] |
| 08:07 |
<_joe_> |
installing the new HHVM package on the api canaries |
[production] |
| 08:04 |
<moritzm> |
repooled mw1221-mw1229 (for T104968) |
[production] |
| 07:53 |
<moritzm> |
depooled mw1221-mw1229 (for T104968) |
[production] |
| 07:49 |
<moritzm> |
repooled mw1230-mw1235 (for T104968) |
[production] |
| 07:43 |
<_joe_> |
installing the new hhvm package on the canary appservers |
[production] |
| 07:08 |
<moritzm> |
depooled mw1230-mw1235 (for T104968) |
[production] |
| 04:31 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Mon Sep 21 04:31:03 UTC 2015 (duration 31m 2s) |
[production] |
| 02:23 |
<l10nupdate@tin> |
LocalisationUpdate completed (1.26wmf23) at 2015-09-21 02:23:12+00:00 |
[production] |
| 02:19 |
<l10nupdate@tin> |
Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 25s) |
[production] |
| 02:06 |
<MaxSem> |
Maps: created indexes on admin. <3 Postgres :( |
[production] |
| 01:56 |
<bblack> |
downtimed eqiad ipv6 text/upload alerts as well, as with mobile above ( 1 301 TLS Redirect - 505 bytes in 1.008 second response time |
[production] |
| 01:46 |
<bblack> |
downtimed the "LVS HTTP IPv6 on mobile-lb.eqiad.wikimedia.org_ipv6" alert for now ( https://phabricator.wikimedia.org/T113154 ) |
[production] |
|
2015-09-19
§
|
| 23:12 |
<urandom> |
begining Cassandra repair on restbase1005 (nodetool repair -pr) |
[production] |
| 23:08 |
<urandom> |
begining Cassandra repair on restbase1004 (nodetool repair -pr) |
[production] |
| 19:56 |
<jynus> |
restarting once more giblit, last chance |
[production] |
| 19:04 |
<paravoid> |
salt rm /etc/systemd/system/txstatsd.service from all cp*, leftover because of ::txstatsd::decommission (removed with 4a1d4e) missing it |
[production] |
| 18:45 |
<_joe_> |
restarted gitblit. I will now substitute myself with a clever perl one-liner. |
[production] |
| 18:38 |
<paravoid> |
pooling back cp1046 to pybal eqiad/mobile, has stayed stable |
[production] |
| 18:34 |
<paravoid> |
reactivating ΒGP with GTT @ eqiad |
[production] |
| 08:42 |
<_joe_> |
cp1046 dead on console again, powercycling to inspect it |
[production] |
| 05:49 |
<aaron@tin> |
Synchronized php-1.26wmf23/extensions/TitleBlacklist: 80d3a21a51f9c54ed2d94 (duration: 00m 12s) |
[production] |
| 05:22 |
<paravoid> |
pybal-depooling cp1046 from eqiad/mobile until further investigation |
[production] |
| 05:21 |
<paravoid> |
powercycling cp1046, dead on console |
[production] |
| 04:28 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sat Sep 19 04:28:59 UTC 2015 (duration 28m 57s) |
[production] |
| 02:23 |
<l10nupdate@tin> |
LocalisationUpdate completed (1.26wmf23) at 2015-09-19 02:23:29+00:00 |
[production] |
| 02:20 |
<l10nupdate@tin> |
Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 05s) |
[production] |
|
2015-09-18
§
|
| 20:52 |
<ebernhardson> |
restart es on elastic1025 to disable dynamic scripting |
[production] |
| 20:34 |
<gwicke> |
dropped by_ns indexes on restbase title_revisions tables |
[production] |
| 19:54 |
<gwicke> |
finished deploy of restbase daacf4daa |
[production] |
| 19:45 |
<gwicke> |
re-enabled puppet on restbase100* |
[production] |
| 19:35 |
<gwicke> |
canary deploy of restbase daacf4daa on restbase1001; moving forward so that we can re-enable puppet over the weekend. |
[production] |
| 17:54 |
<ebernhardson@tin> |
Synchronized wmf-config/CommonSettings.php: Replace insecure es usage with usage of a plugin (duration: 00m 12s) |
[production] |
| 16:41 |
<mutante> |
mailman now on 2.1.18 and jessie |
[production] |
| 16:14 |
<dcausse> |
elastic in eqiad plugin updates: restarting elastic1021 |
[production] |
| 16:07 |
<paravoid> |
deactivating ΒGP with GTT @ eqiad |
[production] |
| 15:20 |
<godog> |
create restbase user on cassandra test cluster |
[production] |
| 14:55 |
<dcausse> |
elastic in eqiad plugin updates: restarting elastic1020 |
[production] |
| 14:22 |
<bblack> |
committing lvs1007-1012 port/vlan changes for asw-d-eqiad (but leaving all 6 LVS ports in "disabled" state - T112781 ) |
[production] |
| 14:14 |
<bblack> |
committing lvs1007-12 port/vlan changes for asw-b-eqiad, round 3... |
[production] |
| 14:11 |
<mutante> |
sodium - stopped exim - rsyncing lists to fermium |
[production] |