301-350 of 10000 results (28ms)
2015-12-14 §
21:11 <subbu> restarted parsoid on wtp1004 (~4 mins back) as a canary [production]
21:04 <subbu> starting parsoid deploy [production]
21:01 <robh> neon returned to normal puppet updates. icinga new cert is live. [production]
20:57 <mutante> mw1259 - enabled hyperthreading (logical processor) [production]
20:54 <mutante> rebooting mw1259 for BIOS settting [production]
20:49 <robh> puppet on neon disabled for certificate update [production]
20:25 <yurik> updated graphoid service [production]
20:15 <hashar> scandium restarted zuul-merger [production]
20:00 <jynus> changing master of db2017 to be now db1018, instead of s2-master [production]
19:20 <jynus> rolling restart s2 codfw mysql servers [production]
19:02 <mutante> reboot cygnus [production]
18:37 <jynus> restarting, upgrading and reconfiguring mysql on db1018 [production]
18:21 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Repool db1057 after maintenance; depool db1018 for maintenance (duration: 00m 29s) [production]
17:20 <thcipriani@tin> Synchronized php-1.27.0-wmf.8/extensions/ContentTranslation/api/ApiContentTranslationToken.php: SWAT: Fix check for JWT [[gerrit:258776]] (duration: 00m 29s) [production]
17:05 <thcipriani@tin> Synchronized php-1.27.0-wmf.8/extensions/UploadWizard/resources/mw.UploadWizardDetails.js: SWAT: mw.UploadWizardDetails update [[gerrit:258767]] (duration: 00m 30s) [production]
16:27 <thcipriani@tin> Synchronized php-1.27.0-wmf.8/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android Beta app: 55% Part II [[gerrit:258985]] (duration: 00m 29s) [production]
16:14 <thcipriani@tin> Synchronized php-1.27.0-wmf.8/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android Beta app: 55% [[gerrit:258985]] (duration: 00m 30s) [production]
16:13 <jynus> performing schema change on testwiki.page (s3) [production]
16:05 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: do not index User namespace on ko.wikipedia [[gerrit:258667]] (duration: 00m 29s) [production]
15:01 <bblack> "service ganglia-monitor restart" on cp*, because most had no varnish stats flowing into ganglia [production]
14:47 <hashar> Stopping zuul-merger daemon on scandium. It lost its disk somehow earlier "DISK CRITICAL - /srv/ssd is not accessible: No such file or directory" [production]
13:31 <bblack> rebooting cp4005 [production]
13:30 <bblack> issues on cp4005, investigating [production]
13:29 <mobrovac> restbase cassandra restarting cassandra on rb1004 due tolow disk space caused by compactions [production]
13:22 <godog> repool restbase1007 [production]
12:04 <mobrovac> restbase deploy on rb1007 after re-imaging [production]
11:31 <godog> upgrade diamond to 3.5-5 in eqiad [production]
11:13 <dcausse> elastic in eqiad: resuming writes [production]
10:50 <jynus> performing schema change on wikishared.cx_translations (x1-master) [production]
10:19 <dcausse> elastic in eqiad: restarting elastic1016 (to release deleted filedesc) [production]
10:11 <jynus> restarting mysql at labsdb1004 [production]
10:11 <dcausse> elastic in eqiad: freezing writes (to restart elastic1016) [production]
09:46 <godog> reimage restbase1007 [production]
02:29 <l10nupdate@tin> ResourceLoader cache refresh completed at Mon Dec 14 02:29:39 UTC 2015 (duration 6m 57s) [production]
02:22 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.8) (duration: 09m 00s) [production]
2015-12-13 §
11:10 <dcausse> elastic in eqiad: restarting elastic1012 to release opened log filedesc [production]
10:54 <godog> extend fluorine's storage lv (94% util) lvresize --size +300G --resizefs /dev/mapper/vg0-lv0 [production]
10:24 <dcausse> elastic in eqiad: disabling TRACE indexing slowlog on jawiki_content and zhwiki_content [production]
07:06 <ori> elasticsearch1012 ditto. [production]
07:05 <ori> elasticsearch1016: Moving /var/log/elasticsearch/* to /var/lib/elasticsearch/logs to free up space. [production]
02:28 <l10nupdate@tin> ResourceLoader cache refresh completed at Sun Dec 13 02:28:52 UTC 2015 (duration 6m 49s) [production]
02:22 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.8) (duration: 09m 08s) [production]
2015-12-12 §
21:26 <legoktm> running fixDefaultJsonContentPages.php on all wikis (T108663) [production]
21:02 <ori@tin> Synchronized wmf-config/jobqueue-eqiad.php: I53f13a159: Add rdb100[5-6] to job queue configuration (duration: 00m 31s) [production]
19:30 <hashar> Restarted Zuul [production]
17:01 <krenair@tin> Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/258669/ (duration: 00m 29s) [production]
10:12 <godog> bounce nslcd on tools-submit and stop puppet [production]
10:00 <dcausse> elastic in eqiad: disabling TRACE indexing slowlog for urwiki_content [production]
09:44 <jynus> recreated events on db1057 with sql_bin_log = 0 and restarted replication on db2016 [production]
09:15 <godog> move old elasticsearch logs on elastic1026 out to /var/lib/elasticsearch/log (/ is full) [production]