|
2019-04-26
§
|
| 17:50 |
<mutante> |
analytics1052 - reported broken systemd state in Icinga - service mcelog was in state failed - systemctl start mcelog - (T212219 ?) |
[production] |
| 16:18 |
<jynus> |
stop s6 mariadb instance on dbstore2001 T220572 |
[production] |
| 16:18 |
<xSavitar> |
Restart successful, tool up and running again |
[tools.awmd-stats] |
| 16:14 |
<xSavitar> |
Restarting the tool |
[tools.awmd-stats] |
| 15:51 |
<arturo> |
andrew updated dns servers for the cloud-instances2-b-eqiad subnet in neutron: 208.80.154.143 and 208.80.154.24 |
[admin] |
| 15:34 |
<jbond42> |
upgrade puppet 4=> 5 and facter 2 => 3 on canary hosts: thumbor1001 ms-fe1005 ms-be1013 scb1001 restbase1007 |
[production] |
| 15:27 |
<chicocvenancio> |
paws-proxy-02 redirecting main page to https://www.mediawiki.org/wiki/PAWS T221886 |
[paws] |
| 15:05 |
<jbond42> |
upgrade puppet 4=> 5 and facter 2 => 3 on canary hosts: ores1001.yaml wtp1025.yaml rdb1006.yaml |
[production] |
| 14:18 |
<marostegui> |
Set pc1004-1006 and pc2004-2006 as unracked on netbox - T209858 T210969 |
[production] |
| 13:33 |
<Krenair> |
shut off deployment-conf03 after discussion with otto.mata and elu.key - it seems ancient, broken, unused. T218729 |
[releng] |
| 13:17 |
<jbond42> |
upgrade puppet 4=> 5 and facter 2 => 3 on canary hosts: mw1311.yaml, mx2001 & dubnium |
[production] |
| 12:52 |
<ema> |
cp4025: restart varnish-be due to mbox lag |
[production] |
| 12:50 |
<jijiki> |
Restarting hhvm on mw1288 |
[production] |
| 12:48 |
<jbond42> |
upgrade puppet 4=> 5 and facter 2 => 3 on mc1019, maps1001 and logstash1007 |
[production] |
| 12:45 |
<ema@puppetmaster1001> |
conftool action : set/pooled=yes; selector: cluster=cache_upload,name=cp4021.ulsfo.wmnet,dc=ulsfo |
[production] |
| 12:44 |
<ema> |
pool cp4021 w/ ATS backend T219967 |
[production] |
| 12:37 |
<andrewbogott> |
test log |
[testlabs] |
| 12:20 |
<andrewbogott> |
rescheduling every pod everywhere |
[tools] |
| 12:20 |
<ema> |
repool cp3030 after directors.frontend.vcl testing T219967 |
[production] |
| 12:18 |
<andrewbogott> |
rescheduling all pods on tools-worker-1023.tools.eqiad.wmflabs |
[tools] |
| 12:09 |
<jbond42> |
upgrade puppet 4=> 5 and facter 2 => 3 on canary hosts: elastic1017, ganeti2001, analytics1042 |
[production] |
| 11:26 |
<jbond42> |
upgrade puppet 4=> 5 and facter 2 => 3 on lvs4007, dns2001 and multatuli |
[production] |
| 11:16 |
<jbond42> |
upgrade puppet 4=> 5 and facter 2 => 3 on bast4002, aqs1004 and conf2001 |
[production] |
| 10:53 |
<Lucas_WMDE> |
wikidata-shex added namespace alias for Schema (T221945) |
[wikidata-dev] |
| 10:28 |
<moritzm> |
restarting Parsoid on wtp1025 for glibc update |
[production] |
| 10:19 |
<ema> |
depool cp3030 for testing T219967 |
[production] |
| 09:48 |
<marostegui> |
Remove labtestservices2001 from tendril - T218022 |
[production] |
| 09:11 |
<moritzm> |
restarting AQS on aqs1004 for glibc update |
[production] |
| 08:42 |
<elukey> |
restart pdfrender on scb1003 (alert flapping) |
[production] |
| 08:21 |
<moritzm> |
uploaded php-xdebug 2.7.0+wmf1 for component/php72 (T221923) |
[production] |
| 07:20 |
<moritzm> |
installing glibc updates on a number of analytics hosts |
[production] |
| 04:58 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1113:3316 T221782 (duration: 00m 56s) |
[production] |
| 02:14 |
<bd808> |
Manually fixed broken apt state on wikimedia-ui.design.eqiad.wmflabs which was preventing puppet from running |
[design] |
| 00:31 |
<eileen> |
civicrm revision changed from 88736c7c11 to 34027da7df, config revision is 2119df9495 |
[production] |
|
2019-04-25
§
|
| 23:41 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 23:41 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 23:41 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 23:41 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 23:39 |
<eileen> |
civicrm revision changed from 519fe8028e to 88736c7c11, config revision is 2119df9495 - deployed patch to start recording payment_processor_id on recurring |
[production] |
| 22:57 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 22:56 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 22:56 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 22:56 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 21:31 |
<andrewbogott> |
stopping nova services on labnet1001/1002 |
[production] |
| 21:26 |
<andrewbogott> |
revoking M5 grants as per https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/506428/4/modules/role/templates/mariadb/grants/production-m5.sql.erb and https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/506345/3/modules/role/templates/mariadb/grants/production-m5.sql.erb |
[production] |
| 21:12 |
<tgr> |
T221516 running mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=metawiki --logwiki=metawiki 'FoldDownPro' 'MichaelOBFDP' |
[production] |
| 19:45 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@7187e0c]: Bump HTML content version in docs, remove Parsoid stash fall-back and start logging all sections requests - T221432 T215956 T216636 (duration: 20m 04s) |
[production] |
| 19:25 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@7187e0c]: Bump HTML content version in docs, remove Parsoid stash fall-back and start logging all sections requests - T221432 T215956 T216636 |
[production] |
| 19:24 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@7187e0c] (dev-cluster): Bump HTML content version in docs and remove Parsoid stash fall-back (duration: 03m 10s) |
[production] |
| 19:21 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@7187e0c] (dev-cluster): Bump HTML content version in docs and remove Parsoid stash fall-back |
[production] |