|
2019-04-26
§
|
| 20:15 |
<foks> |
changing email and password for "Lemon martini@global" |
[production] |
| 19:38 |
<foks> |
changing password for JDiPierro@global |
[production] |
| 19:21 |
<bblack> |
varnish-backend-restart on cp4026, evidence of artificial 503s from mbox lag behavior, probably related to the semi-abuse client doing odd 404 traffic to ulsfo that's triggering bugs in swift's rewrite.py .... |
[production] |
| 19:04 |
<foks> |
changing password for Subinsebastien |
[production] |
| 17:50 |
<mutante> |
analytics1052 - reported broken systemd state in Icinga - service mcelog was in state failed - systemctl start mcelog - (T212219 ?) |
[production] |
| 16:18 |
<jynus> |
stop s6 mariadb instance on dbstore2001 T220572 |
[production] |
| 15:34 |
<jbond42> |
upgrade puppet 4=> 5 and facter 2 => 3 on canary hosts: thumbor1001 ms-fe1005 ms-be1013 scb1001 restbase1007 |
[production] |
| 15:05 |
<jbond42> |
upgrade puppet 4=> 5 and facter 2 => 3 on canary hosts: ores1001.yaml wtp1025.yaml rdb1006.yaml |
[production] |
| 14:18 |
<marostegui> |
Set pc1004-1006 and pc2004-2006 as unracked on netbox - T209858 T210969 |
[production] |
| 13:17 |
<jbond42> |
upgrade puppet 4=> 5 and facter 2 => 3 on canary hosts: mw1311.yaml, mx2001 & dubnium |
[production] |
| 12:52 |
<ema> |
cp4025: restart varnish-be due to mbox lag |
[production] |
| 12:50 |
<jijiki> |
Restarting hhvm on mw1288 |
[production] |
| 12:48 |
<jbond42> |
upgrade puppet 4=> 5 and facter 2 => 3 on mc1019, maps1001 and logstash1007 |
[production] |
| 12:45 |
<ema@puppetmaster1001> |
conftool action : set/pooled=yes; selector: cluster=cache_upload,name=cp4021.ulsfo.wmnet,dc=ulsfo |
[production] |
| 12:44 |
<ema> |
pool cp4021 w/ ATS backend T219967 |
[production] |
| 12:20 |
<ema> |
repool cp3030 after directors.frontend.vcl testing T219967 |
[production] |
| 12:09 |
<jbond42> |
upgrade puppet 4=> 5 and facter 2 => 3 on canary hosts: elastic1017, ganeti2001, analytics1042 |
[production] |
| 11:26 |
<jbond42> |
upgrade puppet 4=> 5 and facter 2 => 3 on lvs4007, dns2001 and multatuli |
[production] |
| 11:16 |
<jbond42> |
upgrade puppet 4=> 5 and facter 2 => 3 on bast4002, aqs1004 and conf2001 |
[production] |
| 10:28 |
<moritzm> |
restarting Parsoid on wtp1025 for glibc update |
[production] |
| 10:19 |
<ema> |
depool cp3030 for testing T219967 |
[production] |
| 09:48 |
<marostegui> |
Remove labtestservices2001 from tendril - T218022 |
[production] |
| 09:11 |
<moritzm> |
restarting AQS on aqs1004 for glibc update |
[production] |
| 08:42 |
<elukey> |
restart pdfrender on scb1003 (alert flapping) |
[production] |
| 08:21 |
<moritzm> |
uploaded php-xdebug 2.7.0+wmf1 for component/php72 (T221923) |
[production] |
| 07:20 |
<moritzm> |
installing glibc updates on a number of analytics hosts |
[production] |
| 04:58 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1113:3316 T221782 (duration: 00m 56s) |
[production] |
| 00:31 |
<eileen> |
civicrm revision changed from 88736c7c11 to 34027da7df, config revision is 2119df9495 |
[production] |
|
2019-04-25
§
|
| 23:41 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 23:41 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 23:41 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 23:41 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 23:39 |
<eileen> |
civicrm revision changed from 519fe8028e to 88736c7c11, config revision is 2119df9495 - deployed patch to start recording payment_processor_id on recurring |
[production] |
| 22:57 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 22:56 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 22:56 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 22:56 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 21:31 |
<andrewbogott> |
stopping nova services on labnet1001/1002 |
[production] |
| 21:26 |
<andrewbogott> |
revoking M5 grants as per https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/506428/4/modules/role/templates/mariadb/grants/production-m5.sql.erb and https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/506345/3/modules/role/templates/mariadb/grants/production-m5.sql.erb |
[production] |
| 21:12 |
<tgr> |
T221516 running mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=metawiki --logwiki=metawiki 'FoldDownPro' 'MichaelOBFDP' |
[production] |
| 19:45 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@7187e0c]: Bump HTML content version in docs, remove Parsoid stash fall-back and start logging all sections requests - T221432 T215956 T216636 (duration: 20m 04s) |
[production] |
| 19:25 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@7187e0c]: Bump HTML content version in docs, remove Parsoid stash fall-back and start logging all sections requests - T221432 T215956 T216636 |
[production] |
| 19:24 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@7187e0c] (dev-cluster): Bump HTML content version in docs and remove Parsoid stash fall-back (duration: 03m 10s) |
[production] |
| 19:21 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@7187e0c] (dev-cluster): Bump HTML content version in docs and remove Parsoid stash fall-back |
[production] |
| 18:32 |
<sbisson@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:506316|Cleanup old EchoCrossWikiBetaFeature]] (2/2) (duration: 00m 53s) |
[production] |
| 18:31 |
<sbisson@deploy1001> |
Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:506316|Cleanup old EchoCrossWikiBetaFeature]] (1/2) (duration: 00m 54s) |
[production] |
| 18:24 |
<sbisson@deploy1001> |
Synchronized php-1.34.0-wmf.1/extensions/GrowthExperiments/includes/EventLogging/SpecialHomepageLogger.php: SWAT: [[gerrit:506210|EventLogging: Make namespace int, use enum for impact module state]] (duration: 00m 54s) |
[production] |
| 16:58 |
<XioNoX> |
add analytics firewall filter term schema to cr1/2-eqiad - T221690 |
[production] |
| 16:57 |
<XioNoX> |
reorganize analytics firewall filters terms (description) on cr1/2-eqiad |
[production] |
| 16:34 |
<moritzm> |
rolling restart of Cassandra on restbase1016-1018 to pick up Java security update |
[production] |