|
2016-11-07
§
|
| 15:01 |
<hashar> |
T150166 mwscript sql.php --wiki=tcywiki /srv/mediawiki/php-1.29.0-wmf.1/extensions/ShortUrl/schemas/shorturls.sql |
[production] |
| 15:00 |
<mark> |
Deactivate cr1-eqiad BGP peering with pfw1-eqiad |
[production] |
| 14:49 |
<hashar> |
terbium: scap pull to add shortUrl tables to bdwikimedia and tcywiki |
[production] |
| 14:42 |
<hashar> |
fawiki: renaming user group 'autopatrol' to 'autopatrolled' for T139246 and T144699 with: mwscript migrateUserGroup.php --wiki=fawiki 'autopatrol' 'autopatrolled' |
[production] |
| 14:42 |
<hashar> |
fawiki Done! 417 users in group 'autopatrol' are now in 'autopatrolled' instead. |
[production] |
| 14:40 |
<hashar@tin> |
Synchronized wmf-config/InitialiseSettings.php: Rename 'autopatrol' to 'autopatrolled' on fawiki - T144699 T139246 (duration: 00m 47s) |
[production] |
| 14:33 |
<gehel> |
reboot maps-test* for kernel upgrade |
[production] |
| 14:30 |
<hashar@tin> |
Synchronized wmf-config: (no message) (duration: 00m 53s) |
[production] |
| 14:10 |
<hashar@tin> |
Synchronized php-1.29.0-wmf.1/extensions/Kartographer/extension.json: Fix monobook <maplink> (missing debounce dep) T145521 (duration: 00m 47s) |
[production] |
| 14:01 |
<gilles> |
Pointing deployment-imagescaler01.eqiad.wmflabs' puppet to puppetmaster.thumbor.eqiad.wmflabs |
[releng] |
| 13:56 |
<gehel> |
reboot wdqs1* for kernel upgrade |
[production] |
| 13:52 |
<bblack> |
depooling cp4018 nginx+varnish-fe services for debugging |
[production] |
| 13:36 |
<gehel> |
reboot wdqs2* for kernel upgrade |
[production] |
| 13:34 |
<hashar> |
Flushed nodepool instances. It is bringing up fresh one now. |
[production] |
| 13:26 |
<moritzm> |
rebooting labnodepool1001 for kernel update |
[production] |
| 13:19 |
<hashar> |
shutting down Nodepool (labnodepool1001.eqiad.wmnet reboot) |
[production] |
| 13:06 |
<moritzm> |
rebooting scandium for kernel update |
[production] |
| 12:09 |
<jynus> |
performing schema change on s6 (imagelinks) T139090 |
[production] |
| 12:00 |
<moritzm> |
rebooting wtp1001 for kernel update |
[production] |
| 11:40 |
<ema> |
cp3043: repool varnish-be and varnish-be-rand (T149881) |
[production] |
| 11:33 |
<moritzm> |
rebooting cassandra test hosts (cerium, praseodymium, xenon) for kernel update |
[production] |
| 10:49 |
<moritzm> |
rebooting mw1017/mw1099 for kernel update |
[production] |
| 10:26 |
<moritzm> |
rebooting cp1008 for kernel update |
[production] |
| 10:19 |
<moritzm> |
rebooting bast4001 for kernel update |
[production] |
| 10:07 |
<jynus> |
performing schema change on s5 (imagelinks) T139090 |
[production] |
| 09:45 |
<elukey> |
started 0022558-161020124223818-oozie-oozi-C to rerun wf-text-2016-11-7-07 |
[analytics] |
| 08:46 |
<moritzm> |
uploaded linux-meta 1.11 to carbon (pointing to the new Linux ABI package) |
[production] |
| 08:44 |
<marostegui> |
stopping mysql on db2042 - maintenance- T149553 |
[production] |
| 08:39 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Depool db2042 for maintenance - T149553 (duration: 00m 50s) |
[production] |
| 08:30 |
<marostegui> |
Deploy schema change on s4 master (db2019) commonswiki.revision - T147305 |
[production] |
| 08:00 |
<elukey> |
started 0022441-161020124223818-oozie-oozi-C to rerun wf-text-2016-11-7-04 -> 06 |
[analytics] |
| 07:02 |
<_joe_> |
removing old logfiles on logstash hosts |
[production] |
| 04:53 |
<joal> |
started 0022249-161020124223818-oozie-oozi-C to rerun wf-text-2016-11-7-00 -> 03 |
[analytics] |
| 02:21 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Mon Nov 7 02:21:02 UTC 2016 (duration 4m 18s) |
[production] |
| 02:16 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.1) (duration: 05m 39s) |
[production] |
|
2016-11-05
§
|
| 21:40 |
<bd808> |
Deleted huge logstash1003:/var/log/logstash/logstash.log.1 log file; disk full |
[production] |
| 21:39 |
<bd808> |
Deleted huge logstash1002:/var/log/logstash/logstash.log.1 log file; disk full |
[production] |
| 21:36 |
<bd808@tin> |
Synchronized wmf-config/InitialiseSettings.php: logstash: Temporarily disable EventBus channel (T150106) (duration: 00m 50s) |
[production] |
| 19:58 |
<Zppix> |
merged https://gerrit.wikimedia.org/r/#/c/319908/ |
[tools.lolrrit-wm] |
| 19:54 |
<bd808> |
ELK stack problems are related to Elasticsearch index mapping. Some events are being rejected for not matching the expected mappings and that is filling up the disk on the logstash injestion hosts |
[production] |
| 19:48 |
<Zppix> |
removing whitelist.js from bot files via shell because gerrit change 319983 basically implements the ideas behind it |
[tools.lolrrit-wm] |
| 19:45 |
<bd808> |
Forced several puppet runs on logstash1001 until things stopped changing; out of disk seemed to have messed up apt upgrades |
[production] |
| 19:45 |
<paladox> |
deploying https://gerrit.wikimedia.org/r/#/c/319983/ |
[tools.lolrrit-wm] |