8451-8500 of 10000 results (70ms)
2019-04-29 §
08:33 <elukey> restart keyholder on deploy1001 + rearm keys [production]
08:28 <elukey> restart keyholder-proxy on deploy1001 (attempt to see if new analytics scap settings got applied) [production]
08:25 <oblivian@deploy1001> Synchronized wmf-config/CommonSettings.php: Enable unicode overrides table for php 7.2 T219279 (duration: 00m 53s) [production]
08:25 <jynus> stop dbstore2001:s2 for cloning to db2098 T220572 [production]
08:23 <oblivian@deploy1001> Synchronized wmf-config/Php72ToUpper.php: Adding unicode overrides table for php 7.2 T219279 (duration: 00m 54s) [production]
08:23 <joal@deploy1001> Started deploy [analytics/refinery@53a4eee]: Test of deploy-user change (analytics -> analytics-deploy) [production]
07:58 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Move db2045 from s8 to x1 T219493 (duration: 00m 55s) [production]
07:47 <marostegui> Stop mysql on db2034 (lag will happen on x1 codfw) - T219493 [production]
07:44 <marostegui> Stop replication on db2034 (x1 master) for maintenance - T219493 [production]
07:13 <moritzm> updated stretch netboot image for 9.9 point release [production]
2019-04-28 §
17:46 <jiji@cumin1001> conftool action : set/pooled=no; selector: name=cp3037.esams.wmnet [production]
17:46 <jijiki> Depooling cp3037 [production]
14:55 <James_F> Updated trwiki's MediaWiki:Common.css to not over-ride the logo. [production]
14:53 <James_F> Manually purged the trwiki logos from Varnish as part of updating them for 2 year anniversary. [production]
14:47 <jforrester@deploy1001> Synchronized static/images/project-logos/trwiki.png: trwiki: Update logo for 2 year anniversary, part III (duration: 00m 53s) [production]
14:45 <jforrester@deploy1001> Synchronized static/images/project-logos/trwiki-1.5x.png: trwiki: Update logo for 2 year anniversary, part II (duration: 00m 53s) [production]
14:44 <jforrester@deploy1001> Synchronized static/images/project-logos/trwiki-2x.png: trwiki: Update logo for 2 year anniversary, part I (duration: 00m 55s) [production]
2019-04-27 §
17:44 <elukey> restart pdfrender on scb1002 (alert flapping) [production]
12:37 <jynus> correcting last log, stopping dbstore2002:s1 to clone it to db2097 T220572 [production]
12:37 <jynus> stopping dbstore2002:s6 to clone it to db2097 T220572 [production]
00:11 <foks> reset passwords for FritzSolms@global and Seanhood@global [production]
2019-04-26 §
20:15 <foks> changing email and password for "Lemon martini@global" [production]
19:38 <foks> changing password for JDiPierro@global [production]
19:21 <bblack> varnish-backend-restart on cp4026, evidence of artificial 503s from mbox lag behavior, probably related to the semi-abuse client doing odd 404 traffic to ulsfo that's triggering bugs in swift's rewrite.py .... [production]
19:04 <foks> changing password for Subinsebastien [production]
17:50 <mutante> analytics1052 - reported broken systemd state in Icinga - service mcelog was in state failed - systemctl start mcelog - (T212219 ?) [production]
16:18 <jynus> stop s6 mariadb instance on dbstore2001 T220572 [production]
15:34 <jbond42> upgrade puppet 4=> 5 and facter 2 => 3 on canary hosts: thumbor1001 ms-fe1005 ms-be1013 scb1001 restbase1007 [production]
15:05 <jbond42> upgrade puppet 4=> 5 and facter 2 => 3 on canary hosts: ores1001.yaml wtp1025.yaml rdb1006.yaml [production]
14:18 <marostegui> Set pc1004-1006 and pc2004-2006 as unracked on netbox - T209858 T210969 [production]
13:17 <jbond42> upgrade puppet 4=> 5 and facter 2 => 3 on canary hosts: mw1311.yaml, mx2001 & dubnium [production]
12:52 <ema> cp4025: restart varnish-be due to mbox lag [production]
12:50 <jijiki> Restarting hhvm on mw1288 [production]
12:48 <jbond42> upgrade puppet 4=> 5 and facter 2 => 3 on mc1019, maps1001 and logstash1007 [production]
12:45 <ema@puppetmaster1001> conftool action : set/pooled=yes; selector: cluster=cache_upload,name=cp4021.ulsfo.wmnet,dc=ulsfo [production]
12:44 <ema> pool cp4021 w/ ATS backend T219967 [production]
12:20 <ema> repool cp3030 after directors.frontend.vcl testing T219967 [production]
12:09 <jbond42> upgrade puppet 4=> 5 and facter 2 => 3 on canary hosts: elastic1017, ganeti2001, analytics1042 [production]
11:26 <jbond42> upgrade puppet 4=> 5 and facter 2 => 3 on lvs4007, dns2001 and multatuli [production]
11:16 <jbond42> upgrade puppet 4=> 5 and facter 2 => 3 on bast4002, aqs1004 and conf2001 [production]
10:28 <moritzm> restarting Parsoid on wtp1025 for glibc update [production]
10:19 <ema> depool cp3030 for testing T219967 [production]
09:48 <marostegui> Remove labtestservices2001 from tendril - T218022 [production]
09:11 <moritzm> restarting AQS on aqs1004 for glibc update [production]
08:42 <elukey> restart pdfrender on scb1003 (alert flapping) [production]
08:21 <moritzm> uploaded php-xdebug 2.7.0+wmf1 for component/php72 (T221923) [production]
07:20 <moritzm> installing glibc updates on a number of analytics hosts [production]
04:58 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1113:3316 T221782 (duration: 00m 56s) [production]
00:31 <eileen> civicrm revision changed from 88736c7c11 to 34027da7df, config revision is 2119df9495 [production]
2019-04-25 §
23:41 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]