|
2016-11-07
§
|
| 15:07 |
<marostegui> |
Enabling gtid_domain_id db1020 (m2 master) - T149418 |
[production] |
| 15:07 |
<mark> |
Reactivate cr1-eqiad BGP peering with pfw1-eqiad |
[production] |
| 15:05 |
<mark> |
Chris moved cr1-eqiad:xe-5/0/3 to xe-3/3/2 |
[production] |
| 15:03 |
<hashar> |
T146014 mwscript extensions/ShortUrl/populateShortUrlTable.php --wiki=bdwikimedia (714 titles done) |
[production] |
| 15:02 |
<hashar> |
T150166 mwscript extensions/ShortUrl/populateShortUrlTable.php --wiki=tcywiki (1569 titles done) |
[production] |
| 15:02 |
<moritzm> |
rebooting mw1261-mw1265 (canary app servers) for kernel update |
[production] |
| 15:01 |
<hashar> |
T146014 mwscript sql.php --wiki=bdwikimedia /srv/mediawiki/php-1.29.0-wmf.1/extensions/ShortUrl/schemas/shorturls.sql |
[production] |
| 15:01 |
<hashar> |
T150166 mwscript sql.php --wiki=tcywiki /srv/mediawiki/php-1.29.0-wmf.1/extensions/ShortUrl/schemas/shorturls.sql |
[production] |
| 15:00 |
<mark> |
Deactivate cr1-eqiad BGP peering with pfw1-eqiad |
[production] |
| 14:49 |
<hashar> |
terbium: scap pull to add shortUrl tables to bdwikimedia and tcywiki |
[production] |
| 14:42 |
<hashar> |
fawiki: renaming user group 'autopatrol' to 'autopatrolled' for T139246 and T144699 with: mwscript migrateUserGroup.php --wiki=fawiki 'autopatrol' 'autopatrolled' |
[production] |
| 14:42 |
<hashar> |
fawiki Done! 417 users in group 'autopatrol' are now in 'autopatrolled' instead. |
[production] |
| 14:40 |
<hashar@tin> |
Synchronized wmf-config/InitialiseSettings.php: Rename 'autopatrol' to 'autopatrolled' on fawiki - T144699 T139246 (duration: 00m 47s) |
[production] |
| 14:33 |
<gehel> |
reboot maps-test* for kernel upgrade |
[production] |
| 14:30 |
<hashar@tin> |
Synchronized wmf-config: (no message) (duration: 00m 53s) |
[production] |
| 14:10 |
<hashar@tin> |
Synchronized php-1.29.0-wmf.1/extensions/Kartographer/extension.json: Fix monobook <maplink> (missing debounce dep) T145521 (duration: 00m 47s) |
[production] |
| 13:56 |
<gehel> |
reboot wdqs1* for kernel upgrade |
[production] |
| 13:52 |
<bblack> |
depooling cp4018 nginx+varnish-fe services for debugging |
[production] |
| 13:36 |
<gehel> |
reboot wdqs2* for kernel upgrade |
[production] |
| 13:34 |
<hashar> |
Flushed nodepool instances. It is bringing up fresh one now. |
[production] |
| 13:26 |
<moritzm> |
rebooting labnodepool1001 for kernel update |
[production] |
| 13:19 |
<hashar> |
shutting down Nodepool (labnodepool1001.eqiad.wmnet reboot) |
[production] |
| 13:06 |
<moritzm> |
rebooting scandium for kernel update |
[production] |
| 12:09 |
<jynus> |
performing schema change on s6 (imagelinks) T139090 |
[production] |
| 12:00 |
<moritzm> |
rebooting wtp1001 for kernel update |
[production] |
| 11:40 |
<ema> |
cp3043: repool varnish-be and varnish-be-rand (T149881) |
[production] |
| 11:33 |
<moritzm> |
rebooting cassandra test hosts (cerium, praseodymium, xenon) for kernel update |
[production] |
| 10:49 |
<moritzm> |
rebooting mw1017/mw1099 for kernel update |
[production] |
| 10:26 |
<moritzm> |
rebooting cp1008 for kernel update |
[production] |
| 10:19 |
<moritzm> |
rebooting bast4001 for kernel update |
[production] |
| 10:07 |
<jynus> |
performing schema change on s5 (imagelinks) T139090 |
[production] |
| 08:46 |
<moritzm> |
uploaded linux-meta 1.11 to carbon (pointing to the new Linux ABI package) |
[production] |
| 08:44 |
<marostegui> |
stopping mysql on db2042 - maintenance- T149553 |
[production] |
| 08:39 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Depool db2042 for maintenance - T149553 (duration: 00m 50s) |
[production] |
| 08:30 |
<marostegui> |
Deploy schema change on s4 master (db2019) commonswiki.revision - T147305 |
[production] |
| 07:02 |
<_joe_> |
removing old logfiles on logstash hosts |
[production] |
| 02:21 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Mon Nov 7 02:21:02 UTC 2016 (duration 4m 18s) |
[production] |
| 02:16 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.1) (duration: 05m 39s) |
[production] |
|
2016-11-05
§
|
| 21:40 |
<bd808> |
Deleted huge logstash1003:/var/log/logstash/logstash.log.1 log file; disk full |
[production] |
| 21:39 |
<bd808> |
Deleted huge logstash1002:/var/log/logstash/logstash.log.1 log file; disk full |
[production] |
| 21:36 |
<bd808@tin> |
Synchronized wmf-config/InitialiseSettings.php: logstash: Temporarily disable EventBus channel (T150106) (duration: 00m 50s) |
[production] |
| 19:54 |
<bd808> |
ELK stack problems are related to Elasticsearch index mapping. Some events are being rejected for not matching the expected mappings and that is filling up the disk on the logstash injestion hosts |
[production] |
| 19:45 |
<bd808> |
Forced several puppet runs on logstash1001 until things stopped changing; out of disk seemed to have messed up apt upgrades |
[production] |
| 19:38 |
<bd808> |
Elasticsearch on logstash1001 won't restart due to missing /etc/elasticsearch/scripts directory |
[production] |
| 19:23 |
<bd808> |
Restarted logstash on logstash1001 |
[production] |
| 19:14 |
<bd808> |
Deleted huge logstash1001:/var/log/logstash/logstash.log.1 log file; disk full and difficult to debug with no free space on / |
[production] |