|
2019-04-30
ยง
|
| 19:43 |
<mutante> |
switched netmon1002/netmon2001 from PHP 7.0 to 7.2 but reverted because LibreNMS still had an issue with it |
[production] |
| 19:40 |
<thcipriani@deploy1001> |
Pruned MediaWiki: 1.33.0-wmf.17 (duration: 10m 11s) |
[production] |
| 19:35 |
<smalyshev@deploy1001> |
Started deploy [wdqs/wdqs@4360316]: Redeploy GUI for fixes T222133, T222129, T222181, T222182 |
[production] |
| 19:27 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 19:27 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster codfw completed |
[production] |
| 19:27 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade production -f eventgate-analytics/eventgate-analytics-codfw-values.yaml --reset-values stable/eventgate-analytics [namespace: eventgate-analytics, clusters: codfw] |
[production] |
| 19:26 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 19:26 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster eqiad completed |
[production] |
| 19:26 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade production -f eventgate-analytics/eventgate-analytics-eqiad-values.yaml --reset-values stable/eventgate-analytics [namespace: eventgate-analytics, clusters: eqiad] |
[production] |
| 19:25 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 19:25 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster staging completed |
[production] |
| 19:25 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging -f eventgate-analytics/eventgate-analytics-staging-values.yaml --reset-values stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 19:24 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade production -f eventgate-analytics/eventgate-analytics-staging-values.yaml --reset-values stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 19:24 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade production -f eventgate-analytics/analytics/eventgate-analytics-staging-values.yaml --reset-values stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 18:40 |
<cdanis> |
running puppet on ms-be201[3,5] to bump replication concurrency T221068 |
[production] |
| 18:24 |
<cdanis> |
running puppet on ms-be2014 to bump replication concurrency T221068 |
[production] |
| 18:09 |
<thcipriani> |
start branchcut for 1.34.0-wmf.3 |
[production] |
| 17:16 |
<bsitzmann@deploy1001> |
Finished deploy [mobileapps/deploy@1f09e44]: Update mobileapps to 142ba30 (T217837) (duration: 04m 16s) |
[production] |
| 17:11 |
<bsitzmann@deploy1001> |
Started deploy [mobileapps/deploy@1f09e44]: Update mobileapps to 142ba30 (T217837) |
[production] |
| 16:57 |
<ayounsi@deploy1001> |
Finished deploy [librenms/librenms@0fd8da6]: Rollback LibreNMS (duration: 00m 09s) |
[production] |
| 16:57 |
<ayounsi@deploy1001> |
Started deploy [librenms/librenms@0fd8da6]: Rollback LibreNMS |
[production] |
| 16:52 |
<arturo> |
merging change to `profile::base` and `::raid` https://gerrit.wikimedia.org/r/c/operations/puppet/+/507357 related to T221225 |
[production] |
| 16:36 |
<ayounsi@deploy1001> |
Finished deploy [librenms/librenms@2094575]: Upgrade LibreNMS to 1.51 - T207706 (duration: 00m 11s) |
[production] |
| 16:36 |
<ayounsi@deploy1001> |
Started deploy [librenms/librenms@2094575]: Upgrade LibreNMS to 1.51 - T207706 |
[production] |
| 16:27 |
<XioNoX> |
upgrade librenms to 1.51 |
[production] |
| 16:26 |
<jbond42> |
upgrade puppet and facter in eqsin |
[production] |
| 16:04 |
<ema> |
pool cp4022 w/ ATS backend T219967 |
[production] |
| 15:58 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 15:58 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 15:58 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 15:58 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 15:45 |
<elukey> |
restart hadoop hdfs namenodes on an-master100[1,2] to pick up new logging settings - T220702 |
[production] |
| 15:18 |
<jynus> |
stop s8 instance on dbstore2001 for cloning to db2100 T220572 |
[production] |
| 15:09 |
<jiji@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Send 1% of anonymous users to PHP7.2 - T219150 (duration: 00m 54s) |
[production] |
| 14:58 |
<jbond42> |
enable-puppet "T220987: global kafaka log shipping - staged rollout (jbond)" |
[production] |
| 14:56 |
<cdanis> |
cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin 'bast3002*' 'run-puppet-agent --enable "filippo prometheus"' |
[production] |
| 14:49 |
<cdanis> |
cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin 'labmon1001*' 'run-puppet-agent --enable "staged rollout T222105 by cdanis"' |
[production] |
| 14:44 |
<jijiki> |
Sending 1% of anonymous users to PHP7.2 - T219150 |
[production] |
| 14:43 |
<cdanis> |
cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin 'bast5001*' 'run-puppet-agent --enable "staged rollout T222105 by cdanis"' |
[production] |
| 14:26 |
<jbond42> |
disable-puppet "T220987: global kafaka log shipping - staged rollout (jbond)" |
[production] |
| 14:24 |
<cdanis> |
cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin 'prometheus2004*' 'run-puppet-agent --enable "staged rollout T222105 by cdanis"' |
[production] |
| 14:17 |
<cdanis> |
cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin 'prometheus2003*' 'run-puppet-agent --enable "staged rollout T222105 by cdanis"' |
[production] |
| 14:15 |
<cdanis> |
cdanis@prometheus1003.eqiad.wmnet ~ % sudo enable-puppet 'cdanis testing original query.max-samples T222105' |
[production] |
| 13:29 |
<cdanis> |
cdanis@prometheus1004.eqiad.wmnet ~ % sudo systemctl restart prometheus@ops.service |
[production] |
| 13:28 |
<ema> |
depool cp4022 and reimage as upload_ats T219967 |
[production] |
| 13:20 |
<arturo> |
reverting sudo puppet module changes https://gerrit.wikimedia.org/r/c/operations/puppet/+/507317 |
[production] |
| 13:16 |
<cdanis> |
cdanis@prometheus1003.eqiad.wmnet ~ % sudo systemctl restart prometheus@ops.service |
[production] |
| 13:15 |
<cdanis> |
cdanis@prometheus1003.eqiad.wmnet ~ % sudo disable-puppet 'cdanis testing original query.max-samples T222105' |
[production] |
| 13:08 |
<cdanis> |
OOMed the eqiad ops prometheus @ prometheus1003 |
[production] |
| 13:02 |
<cdanis> |
OOMed the eqiad ops prometheus @ prometheus1004 |
[production] |