6251-6300 of 10000 results (58ms)
2016-02-15 §
16:34 <bd808> Ran sync-common on mw1119 after stuck scap update was killed there [production]
16:29 <bd808> Killed stuck rsync by mw1119 that was keeping l10nudpate running [production]
16:28 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.13) (duration: 855m 18s) [production]
16:28 <jzerebecki> reloading zuul for 2d16ad3..3bb0afa [releng]
16:27 <moritzm> installing postgres security updates on maps* [production]
16:26 <halfak> restarted workers on ores-worker -02 and -03 [ores]
16:10 <hashar> Image ci-jessie-wikimedia-1455552377 in wmflabs-eqiad is ready [releng]
15:32 <_joe_> restarting the salt minions on all deployment targets [production]
15:25 <jzerebecki> reloading zuul for e174335..2d16ad3 [releng]
15:23 <hashar> Image ci-jessie-wikimedia-1455549539 in wmflabs-eqiad is ready [releng]
15:22 <apergos> disabled puppet again on iridium, had to restore sshd config from filebucket after puppet run [production]
15:19 <hashar> Regenerating Nodepool snapshot. Slave scripts have 0 bytes... [releng]
15:18 <hoo|busy> Changed email for global account "Frau pomerenke" [production]
15:04 <hashar> Slave scripts added to Nodepool instances! Image ci-jessie-wikimedia-1455548346 in wmflabs-eqiad is ready [releng]
14:51 <apergos> phab role commented out on iridium and puppet re-enabled, no phab changes will be applied there [production]
14:24 <godog> start restbase1007-b cassandra instance, bootstrapping T119935 [production]
13:22 <joal> Failed deploying refinery: deploy from tin failed, need ottomata help [analytics]
13:04 <joal> Deploying refinery [analytics]
12:30 <_joe_> deployment master is now tin [production]
11:57 <_joe_> switching the deployment host back to tin [production]
11:48 <_joe_> restarted hhvm on mw1130, memory exhausted [production]
11:28 <jynus> offlining disk 32:5 on db1063 [production]
11:05 <hashar> Image ci-jessie-wikimedia-1455534001 in wmflabs-eqiad is ready [releng]
10:58 <hoo@mira> Synchronized wmf-config/throttle.php: Throttle exception for [[de:WP:Wikimedia Deutschland/WPFF Berlinale2016]] (duration: 01m 28s) [production]
10:57 <hashar> bumped Nodepool disk image to fix apt configuration / include jre and package-config. Refreshing snapshot image. Poke T126246 T126230 T119143 [releng]
09:38 <godog> swift codfw-prod: ms-be2020 / ms-be2021 weight to 3500 [production]
09:16 <gehel> re-enabling puppet on elastic05 (https://phabricator.wikimedia.org/T126891) [deployment-prep]
09:09 <_joe_> powercycled mw1131, OOM'd [production]
07:52 <legoktm> deploying https://gerrit.wikimedia.org/r/270686 [releng]
06:52 <legoktm> legoktm@gallium:/srv/org/wikimedia/doc$ sudo -u jenkins-slave rm -rf EventLogging/ GuidedTour/ MultimediaViewer/ TemplateData/ [releng]
06:22 <legoktm> deploying https://gerrit.wikimedia.org/r/270677 [releng]
06:12 <legoktm> deploying https://gerrit.wikimedia.org/r/270675 [releng]
06:02 <legoktm> deploying https://gerrit.wikimedia.org/r/270674 [releng]
05:56 <legoktm> deploying https://gerrit.wikimedia.org/r/270673 [releng]
05:32 <legoktm> deploying https://gerrit.wikimedia.org/r/270670 [releng]
04:05 <legoktm> deploying https://gerrit.wikimedia.org/r/270667 [releng]
04:01 <halfak> Ammending last message -- Restarted the uwsgi-ores-web service on *ores-web-01*. The service on ores-web-02 was left alone. [ores]
03:59 <halfak> Restarted the uwsgi-ores-web service on ores-web-02 [ores]
03:26 <legoktm> deploying https://gerrit.wikimedia.org/r/270665 [releng]
03:14 <bd808> Cleaned up junk data from before project validation was in place [tools.sal]
02:56 <legoktm> deploying https://gerrit.wikimedia.org/r/270657 [releng]
01:01 <bd808> Upgraded to 64fcdcf; !log project name validation in #wikimedia-labs; fixed bug in Phab mention regex [tools.stashbot]
2016-02-14 §
23:53 <legoktm> deploying https://gerrit.wikimedia.org/r/270656 [releng]
23:25 <legoktm> deploying https://gerrit.wikimedia.org/r/270654 [releng]
23:18 <jynus> repaired table and restarted replication on sanitarium (s3) [production]
23:13 <legoktm> also deploying https://gerrit.wikimedia.org/r/#/c/265098/ [releng]
23:11 <legoktm> deploying https://gerrit.wikimedia.org/r/270651 [releng]
23:06 <andrewbogott> moved tools dbs off of labsdb1002 via https://gerrit.wikimedia.org/r/#/c/270650/ [production]
22:40 <_joe_> labsdb lvm partition /srv unavailable because /dev/sdc is apparently broken [production]
16:55 <_joe_> powercycling mw1140, stuck in OOM, no ssh no console login [production]