3951-4000 of 10000 results (68ms)
2019-05-31 §
07:15 <_joe_> draining mw1348 from traffic [production]
07:14 <jynus> depool labsdb1009 for maintenance [production]
06:55 <jynus> upgrade and restart db2058 [production]
06:33 <_joe_> repooled mw1348 [production]
06:21 <jijiki> depool mw1348 [production]
06:16 <_joe_> restarting php-fpm on mw1348 [production]
00:08 <jgleeson> Updating civicrm from bb4acf3d8a to e028bfcd63 [production]
2019-05-30 §
23:36 <XioNoX> remove BGP sessions to starhub on cr4-ulsfo (left the IXP) [production]
22:59 <marxarelli> deleted 95 docker images from contint1001, freeing ~ 8G on / cc: T219850 [production]
22:59 <XioNoX> add terms to drop specific icmp frag packets from cr1/2-eqiad - T224186 [production]
22:53 <marxarelli> deleting stale docker images from contint1001, cc: T207707 T219850 [production]
22:25 <mutante> phab2001 / phab1003 - why is 'git status' in /srv/phab/phabricator unclean with lots of file deletions but also not identical [production]
22:24 <mutante> phab2001 - scap pull - but it fails with directory /srv/mediawiki not found that's so wrong [production]
22:20 <niharika29@deploy1001> Synchronized php-1.34.0-wmf.7/extensions/WikimediaEvents/: Avoid division by zero warnings T224686 (duration: 00m 49s) [production]
22:19 <niharika29@deploy1001> Synchronized php-1.34.0-wmf.7/extensions/PageTriage/: Fix broken feed - T224693 (duration: 00m 51s) [production]
21:27 <Krinkle> krinkle@mwmaint1002 Add 1 row to pagetriage_tags table on test2wiki db, based on PageTriageTagsPatch-recreated.sql. T224693, T189929 [production]
21:12 <Krinkle> krinkle@mwmaint1002 Add 1 row to pagetriage_tags table on testwiki db, based on PageTriageTagsPatch-recreated.sql. T224693, T189929 [production]
21:11 <Krinkle> krinkle@mwmaint1002 Add 1 row to pagetriage_tags table on enwiki, based on PageTriageTagsPatch-recreated.sql. T224693, T189929 [production]
21:10 <niharika29@deploy1001> Synchronized php-1.34.0-wmf.7/extensions/PageTriage: Bump wgPageTriageCacheVersion T224693 (duration: 00m 51s) [production]
21:07 <XioNoX> add RPKI sessions on cr4-ulsfo - T220669 [production]
20:39 <twentyafterfour> phabricator: restart ssh-phab.service [production]
19:49 <mutante> sodium (mirrors) - sudo -u mirror /usr/local/sbin/update-ubuntu-mirror [production]
18:49 <Urbanecm> Morning SWAT finished [production]
18:47 <urbanecm@deploy1001> Synchronized php-1.34.0-wmf.7/extensions/GrowthExperiments/: [[:gerrit:513300|QuestionPoster: Correctly set timestamp when question is posted]] (T223338) (duration: 00m 51s) [production]
18:26 <mutante> phab1003 - switch 'vcs' user to 'NP' to match phab1001 setup and then /srv/phab/phabricator# ./bin/config set diffusion.ssh-user vcs (T224677) [production]
18:24 <XioNoX> bounce eqord-ulsfo interface to try to fix BFD sessions [production]
18:12 <Krinkle> Running `php7adm /opcache-free` on mw1348 and mw1321, T224491 [production]
18:12 <Krinkle> Running `php7adm /opcache-free` on mw1348 and mw1321 [production]
18:11 <Krinkle> mw1348 (recent api/php72 100% experiment) shows signs of corruption [production]
18:11 <Krinkle> mw1321 php7.2 shows signs of corruption for over 2 hours – https://phabricator.wikimedia.org/T224491#5224464 [production]
18:03 <krinkle@deploy1001> Synchronized wmf-config/arclamp.php: (no justification provided) (duration: 00m 53s) [production]
16:24 <bblack> re-pool cp3047 into service as ats-be - T222937 [production]
16:04 <mutante> phab1001 - removing 2620:0:861:103:10:64:32:186/128 from eth0 [production]
16:03 <mutante> phab1001 - removing 10.64.32.186/32 from eth0 [production]
16:01 <mutante> phab1001 - removing git-ssh.wm.org IP from interface - phab1003 - activating IPv6 listen address for git-ssh [production]
15:36 <jynus> stop es1019 for maintenance T213422 [production]
15:26 <cmjohnson1> shutting down db1099 to swap DIMM T221502 [production]
15:20 <jynus@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1089 with full weight; depool es1019 (duration: 00m 52s) [production]
15:19 <herron> performing rolling reboots of eqiad kafka main cluster hosts for security updates [production]
15:06 <onimisionipe> pooled maps2004 - osm import is complete - T224395 [production]
14:44 <andrewbogott> reimaging cloudvirtan1001 for T224566 [production]
14:43 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:43 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:42 <andrewbogott> reimaging cloudvirtan1001 [production]
14:29 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:29 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:25 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:25 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:22 <bblack> rebooting cp3047 (post-reimage/puppetization for T222937) [production]
14:21 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]