551-600 of 10000 results (26ms)
2016-12-26 §
19:08 <otto@tin> Finished deploy [eventstreams/deploy@e771863]: (no message) (duration: 01m 05s) [production]
19:07 <otto@tin> Starting deploy [eventstreams/deploy@e771863]: (no message) [production]
19:07 <otto@tin> Finished deploy [eventstreams/deploy@e771863]: log (duration: 00m 03s) [production]
19:07 <otto@tin> Starting deploy [eventstreams/deploy@e771863]: log [production]
19:07 <otto@tin> Finished deploy [eventstreams/deploy@e771863]: (no message) (duration: 00m 04s) [production]
19:07 <otto@tin> Starting deploy [eventstreams/deploy@e771863]: (no message) [production]
19:07 <otto@tin> Finished deploy [eventstreams/deploy@e771863]: (no message) (duration: 00m 10s) [production]
19:06 <otto@tin> Starting deploy [eventstreams/deploy@e771863]: (no message) [production]
2016-12-25 §
08:30 <jynus> restarting dbstore1002 mariadb, alive but unresponsive to queries do to long running queries saturating the service [production]
2016-12-23 §
21:29 <mutante> restarted apache on mw canary servers [production]
19:55 <mutante> cobalt - restarted apache [production]
19:46 <mutante> install libxml2 upgrade on bastions and mw-canary [production]
19:38 <mutante> install libxml2 upgrade on "misc-others" [production]
16:32 <robh> rebooted db2060 due to some kind of hw raid or xfs failure, wouldnt take input [production]
15:13 <jynus> testing labsdb-analytics automatic failover [production]
09:23 <jynus> stopping slave dbstore2001(s2) and db2034 for more table cloning [production]
07:56 <moritzm> installing Python security updates [production]
2016-12-22 §
23:16 <ejegg> updated fundraising dashboard from c2cccc058e3e7f929dca5c733c29a3054bca6712 to bec0077599737c774cc716851171365b38a8b02a [production]
22:46 <ejegg> updated fundraising dashboard from ea4305bc9b2c919d3f828ba542eb32b91b207af7 to c2cccc058e3e7f929dca5c733c29a3054bca6712 [production]
22:11 <thcipriani> disable l10nupdate cron for deployment freeze [production]
21:25 <ebernhardson> restarting elasticsearch (again) on relforge100[12] to test ltr plugin [production]
21:10 <RoanKattouw> Got lots of errors like 20:49:56 Unable to find remote tracking branch/tag for /srv/mediawiki-staging/php-1.29.0-wmf.6/extensions/ParserFunctions during this scap [production]
21:03 <catrope@tin> Finished scap: Sync Idf4618977f172 in the OAuth extension- (duration: 24m 16s) [production]
20:39 <catrope@tin> Started scap: Sync Idf4618977f172 in the OAuth extension- [production]
19:59 <ebernhardson> restarting elasticsearch (again) on relforge100[12] to test ltr plugin [production]
19:22 <gehel> restart wdqs-blazegraph and wdqs-updater on wdqs1001.eqiad.wmnet (suspicious load) [production]
19:18 <ebernhardson> restarting elasticsearch on relforge100[12] to test ltr plugin [production]
19:18 <jynus> stopping replication on dbstore2001(s2) and db2035 for enwiktionary.templatelinks reimport [production]
19:04 <godog> roll restart swift proxy on ms-fe1* to drain thumbor traffic [production]
15:39 <jynus> restart dbstore2001 to change buffer pool size, testing gerrit:328671 [production]
14:51 <elukey> restarting the yarn node manager java daemons on all the Hadoop worker nodes due to suspect memory leak [production]
14:14 <elukey> the previous entry is missing: "on analytics1032" [production]
14:13 <elukey> manually starting the yarn nodemanager after OOM [production]
13:41 <jynus> stopping db1035 (depooled) replication to perform maintenance to avoid disk alerts in the next 2 weeks [production]
10:02 <moritzm> installing c-ares security updates on trusty systems (jessie already fixed for quite a while) [production]
10:02 <moritzm> installing c-ares security updates [production]
09:02 <moritzm> installing tomcat security updates [production]
08:45 <moritzm> installing libav security updates on trusty systems [production]
08:18 <moritzm> installing Django security updates [production]
07:26 <elukey> created /var/log/squid3/access.log.1.gz on aluminum to fix cronspam - T132324 [production]
02:26 <l10nupdate@tin> ResourceLoader cache refresh completed at Thu Dec 22 02:26:23 UTC 2016 (duration 4m 49s) [production]
02:21 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.6) (duration: 07m 54s) [production]
2016-12-21 §
23:48 <mutante> europium - jessie reinstall done - powered down until until reclaim (T153918) [production]
23:31 <mutante> europium - re-installing with jessie (T82239) [production]
19:15 <mutante> public1-b-eqiad and public1-c-eqiad are configured to use install1001 as DHCP, all others still use carbon as DHCP | all subnets now use install1001 as TFTP [production]
19:13 <mutante> carbon - re-enabled puppet and DHCP [production]
18:13 <mutante> carbon - temp stopping dhcp server [production]
15:22 <gehel> truncating /var/log/elasticsearch/relforge-eqiad_feature.log on relforge100[12] [production]
15:04 <elukey> removed mongodb* packages from stat1003 after https://gerrit.wikimedia.org/r/328519 [production]
14:54 <moritzm> installing ghostscript security updates on trusty hosts [production]