201-250 of 10000 results (20ms)
2016-02-16 §
10:50 <godog> start swiftrepl commons thumbs for top50 popular size T125791 [production]
10:14 <akosiaris> disable puppet, stop salt-minion on sca100{1,2} [production]
09:52 <hashar> will cut the wmf branches this afternoon starting around 14:00 CET [production]
09:31 <hoo@tin> Synchronized wmf-config/InitialiseSettings-labs.php: (no message) (duration: 00m 58s) [production]
02:34 <l10nupdate@tin> ResourceLoader cache refresh completed at Tue Feb 16 02:34:30 UTC 2016 (duration 8m 34s) [production]
02:25 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.13) (duration: 12m 28s) [production]
2016-02-15 §
21:26 <krinkle@tin> Synchronized w/static.php: (no message) (duration: 00m 58s) [production]
20:55 <twentyafterfour> moved contents of /srv/phab/dumps into /srv/dumps [production]
20:48 <twentyafterfour> backing up /srv/phab to /srv/phab.bak in case it needs to be restored in a hurry. Note: the backup excludes /srv/phab/repos which is backed up separately [production]
20:47 <twentyafterfour> brought iridium up to date with a current puppet run and checked that all repositories are at the correct tag [production]
18:23 <volans> Upgrading db1022 (MariaDB and kernel) already depooled and put in scheduled downtime [production]
16:46 <yurik> Deployed patch for T126897 [production]
16:38 <volans@tin> Synchronized wmf-config/db-eqiad.php: depool db1022 (duration: 00m 58s) [production]
16:34 <bd808> Ran sync-common on mw1119 after stuck scap update was killed there [production]
16:29 <bd808> Killed stuck rsync by mw1119 that was keeping l10nudpate running [production]
16:28 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.13) (duration: 855m 18s) [production]
16:27 <moritzm> installing postgres security updates on maps* [production]
15:32 <_joe_> restarting the salt minions on all deployment targets [production]
15:22 <apergos> disabled puppet again on iridium, had to restore sshd config from filebucket after puppet run [production]
15:18 <hoo|busy> Changed email for global account "Frau pomerenke" [production]
14:51 <apergos> phab role commented out on iridium and puppet re-enabled, no phab changes will be applied there [production]
14:24 <godog> start restbase1007-b cassandra instance, bootstrapping T119935 [production]
12:30 <_joe_> deployment master is now tin [production]
11:57 <_joe_> switching the deployment host back to tin [production]
11:48 <_joe_> restarted hhvm on mw1130, memory exhausted [production]
11:28 <jynus> offlining disk 32:5 on db1063 [production]
10:58 <hoo@mira> Synchronized wmf-config/throttle.php: Throttle exception for [[de:WP:Wikimedia Deutschland/WPFF Berlinale2016]] (duration: 01m 28s) [production]
09:38 <godog> swift codfw-prod: ms-be2020 / ms-be2021 weight to 3500 [production]
09:09 <_joe_> powercycled mw1131, OOM'd [production]
2016-02-14 §
23:18 <jynus> repaired table and restarted replication on sanitarium (s3) [production]
23:06 <andrewbogott> moved tools dbs off of labsdb1002 via https://gerrit.wikimedia.org/r/#/c/270650/ [production]
22:40 <_joe_> labsdb lvm partition /srv unavailable because /dev/sdc is apparently broken [production]
16:55 <_joe_> powercycling mw1140, stuck in OOM, no ssh no console login [production]
13:57 <jynus@mira> Synchronized wmf-config/db-eqiad.php: vslow, dump to db1054; do not send api to db1054; 1036 only rc (duration: 01m 17s) [production]
13:14 <hoo@mira> Synchronized wmf-config/db-eqiad.php: Depool db1021 (duration: 01m 26s) [production]
02:40 <l10nupdate@tin> ResourceLoader cache refresh completed at Sun Feb 14 02:40:27 UTC 2016 (duration 9m 8s) [production]
02:31 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.13) (duration: 12m 01s) [production]
2016-02-13 §
06:30 <urandom> `nodetool stop -- COMPACTION && nodetool cleanup' on restbase1002.eqiad, an abundance of caution (https://phabricator.wikimedia.org/P2612) [production]
04:37 <urandom> (ephemerally) dropping compactor thread count from 10 to 8 on restbase1002.eqiad [production]
01:18 <mutante> omg testing this log feature that logs straight to tickets (T108720) [production]
00:42 <krinkle@mira> Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 01m 14s) [production]
00:28 <krinkle@mira> Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 01m 17s) [production]
00:10 <mutante> ruthenium - restarting parsoid, now works out of /srv/ [production]
2016-02-12 §
23:37 <mutante> ruthenium - moving parsoid path, cleaning up old resources [production]
22:33 <ori@mira> Synchronized php-1.27.0-wmf.13/extensions/MobileFrontend/extension.json: I315628aef3: Don't use 'qlow' for NetSpeed=B (duration: 01m 16s) [production]
21:30 <tgr@mira> Synchronized wmf-config/InitialiseSettings.php: T125455: log session-ip channel to logstash (duration: 01m 17s) [production]
21:22 <ori@mira> Synchronized docroot and w: Ifc5b02cba4: Speed trials: add no-srcset variant (duration: 01m 16s) [production]
19:26 <bd808@mira> Synchronized php-1.27.0-wmf.13/extensions/Disambiguator/Disambiguator.hooks.php: Check for array index existence (7b5f87f) (T126651) (duration: 01m 15s) [production]
19:16 <bd808> Wikis back up thankfully [production]
19:15 <bd808@mira> Synchronized php-1.27.0-wmf.13/includes/session/SessionManager.php: Log multiple IPs using the same session or the same user account (4d8b8ca) (T125455) (duration: 01m 17s) [production]