|
2016-11-07
§
|
| 13:56 |
<gehel> |
reboot wdqs1* for kernel upgrade |
[production] |
| 13:52 |
<bblack> |
depooling cp4018 nginx+varnish-fe services for debugging |
[production] |
| 13:36 |
<gehel> |
reboot wdqs2* for kernel upgrade |
[production] |
| 13:34 |
<hashar> |
Flushed nodepool instances. It is bringing up fresh one now. |
[production] |
| 13:26 |
<moritzm> |
rebooting labnodepool1001 for kernel update |
[production] |
| 13:19 |
<hashar> |
shutting down Nodepool (labnodepool1001.eqiad.wmnet reboot) |
[production] |
| 13:06 |
<moritzm> |
rebooting scandium for kernel update |
[production] |
| 12:09 |
<jynus> |
performing schema change on s6 (imagelinks) T139090 |
[production] |
| 12:00 |
<moritzm> |
rebooting wtp1001 for kernel update |
[production] |
| 11:40 |
<ema> |
cp3043: repool varnish-be and varnish-be-rand (T149881) |
[production] |
| 11:33 |
<moritzm> |
rebooting cassandra test hosts (cerium, praseodymium, xenon) for kernel update |
[production] |
| 10:49 |
<moritzm> |
rebooting mw1017/mw1099 for kernel update |
[production] |
| 10:26 |
<moritzm> |
rebooting cp1008 for kernel update |
[production] |
| 10:19 |
<moritzm> |
rebooting bast4001 for kernel update |
[production] |
| 10:07 |
<jynus> |
performing schema change on s5 (imagelinks) T139090 |
[production] |
| 08:46 |
<moritzm> |
uploaded linux-meta 1.11 to carbon (pointing to the new Linux ABI package) |
[production] |
| 08:44 |
<marostegui> |
stopping mysql on db2042 - maintenance- T149553 |
[production] |
| 08:39 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Depool db2042 for maintenance - T149553 (duration: 00m 50s) |
[production] |
| 08:30 |
<marostegui> |
Deploy schema change on s4 master (db2019) commonswiki.revision - T147305 |
[production] |
| 07:02 |
<_joe_> |
removing old logfiles on logstash hosts |
[production] |
| 02:21 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Mon Nov 7 02:21:02 UTC 2016 (duration 4m 18s) |
[production] |
| 02:16 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.1) (duration: 05m 39s) |
[production] |
|
2016-11-05
§
|
| 21:40 |
<bd808> |
Deleted huge logstash1003:/var/log/logstash/logstash.log.1 log file; disk full |
[production] |
| 21:39 |
<bd808> |
Deleted huge logstash1002:/var/log/logstash/logstash.log.1 log file; disk full |
[production] |
| 21:36 |
<bd808@tin> |
Synchronized wmf-config/InitialiseSettings.php: logstash: Temporarily disable EventBus channel (T150106) (duration: 00m 50s) |
[production] |
| 19:54 |
<bd808> |
ELK stack problems are related to Elasticsearch index mapping. Some events are being rejected for not matching the expected mappings and that is filling up the disk on the logstash injestion hosts |
[production] |
| 19:45 |
<bd808> |
Forced several puppet runs on logstash1001 until things stopped changing; out of disk seemed to have messed up apt upgrades |
[production] |
| 19:38 |
<bd808> |
Elasticsearch on logstash1001 won't restart due to missing /etc/elasticsearch/scripts directory |
[production] |
| 19:23 |
<bd808> |
Restarted logstash on logstash1001 |
[production] |
| 19:14 |
<bd808> |
Deleted huge logstash1001:/var/log/logstash/logstash.log.1 log file; disk full and difficult to debug with no free space on / |
[production] |
| 02:21 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sat Nov 5 02:21:37 UTC 2016 (duration 4m 36s) |
[production] |
| 02:17 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.1) (duration: 05m 25s) |
[production] |
|
2016-11-04
§
|
| 22:43 |
<godog> |
stop puppet on einsteinium and tegment to avoid log spam - T150061 |
[production] |
| 21:14 |
<urandom> |
T133395: Starting user-defined compaction of local_group_wikipedia_T_parsoid_html.data, files la-169018-big-Data.db and la-171488-big-Data.db |
[production] |
| 21:06 |
<godog> |
compress huge daemon.log on einsteinium into /srv/ |
[production] |
| 18:11 |
<moritzm> |
uploaded new jessie linux package based on 4.4.30 to carbon |
[production] |
| 18:01 |
<paravoid> |
moving mc1033-mc1036 from asw-d-eqiad to asw2-d-eqiad |
[production] |
| 17:54 |
<paravoid> |
reactivating cr1-eqiad:ae4 and its subinterfaces (VRRP bug seems to have been worked around) |
[production] |
| 17:44 |
<paravoid> |
moved cr1-eqiad:ae4 links from asw-d-eqiad:ae1 to to asw2-d-eqiad:ae1 |
[production] |
| 16:38 |
<ema> |
upgrading cp4018 (text-ulsfo) to varnish 4 -- T131503 |
[production] |
| 16:22 |
<ema> |
upgrading cp4017 (text-ulsfo) to varnish 4 -- T131503 |
[production] |
| 16:00 |
<ema> |
upgrading cp4016 (text-ulsfo) to varnish 4 -- T131503 |
[production] |
| 15:37 |
<ema> |
upgrading cp4010 (text-ulsfo) to varnish 4 -- T131503 |
[production] |
| 15:36 |
<paravoid> |
set up 4x10G (ae0) links between asw-d-eqiad<->asw2-d-eqiad |
[production] |
| 15:35 |
<marostegui> |
reimage dbstore2002 - T150017 |
[production] |
| 15:20 |
<reedy@tin> |
Synchronized wmf-config/InitialiseSettings.php: Remove wikitech bot group (duration: 00m 47s) |
[production] |