|
2019-04-30
ยง
|
| 15:58 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 15:58 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 15:58 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 15:45 |
<elukey> |
restart hadoop hdfs namenodes on an-master100[1,2] to pick up new logging settings - T220702 |
[production] |
| 15:18 |
<jynus> |
stop s8 instance on dbstore2001 for cloning to db2100 T220572 |
[production] |
| 15:09 |
<jiji@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Send 1% of anonymous users to PHP7.2 - T219150 (duration: 00m 54s) |
[production] |
| 14:58 |
<jbond42> |
enable-puppet "T220987: global kafaka log shipping - staged rollout (jbond)" |
[production] |
| 14:56 |
<cdanis> |
cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin 'bast3002*' 'run-puppet-agent --enable "filippo prometheus"' |
[production] |
| 14:49 |
<cdanis> |
cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin 'labmon1001*' 'run-puppet-agent --enable "staged rollout T222105 by cdanis"' |
[production] |
| 14:44 |
<jijiki> |
Sending 1% of anonymous users to PHP7.2 - T219150 |
[production] |
| 14:43 |
<cdanis> |
cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin 'bast5001*' 'run-puppet-agent --enable "staged rollout T222105 by cdanis"' |
[production] |
| 14:26 |
<jbond42> |
disable-puppet "T220987: global kafaka log shipping - staged rollout (jbond)" |
[production] |
| 14:24 |
<cdanis> |
cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin 'prometheus2004*' 'run-puppet-agent --enable "staged rollout T222105 by cdanis"' |
[production] |
| 14:17 |
<cdanis> |
cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin 'prometheus2003*' 'run-puppet-agent --enable "staged rollout T222105 by cdanis"' |
[production] |
| 14:15 |
<cdanis> |
cdanis@prometheus1003.eqiad.wmnet ~ % sudo enable-puppet 'cdanis testing original query.max-samples T222105' |
[production] |
| 13:29 |
<cdanis> |
cdanis@prometheus1004.eqiad.wmnet ~ % sudo systemctl restart prometheus@ops.service |
[production] |
| 13:28 |
<ema> |
depool cp4022 and reimage as upload_ats T219967 |
[production] |
| 13:20 |
<arturo> |
reverting sudo puppet module changes https://gerrit.wikimedia.org/r/c/operations/puppet/+/507317 |
[production] |
| 13:16 |
<cdanis> |
cdanis@prometheus1003.eqiad.wmnet ~ % sudo systemctl restart prometheus@ops.service |
[production] |
| 13:15 |
<cdanis> |
cdanis@prometheus1003.eqiad.wmnet ~ % sudo disable-puppet 'cdanis testing original query.max-samples T222105' |
[production] |
| 13:08 |
<cdanis> |
OOMed the eqiad ops prometheus @ prometheus1003 |
[production] |
| 13:02 |
<cdanis> |
OOMed the eqiad ops prometheus @ prometheus1004 |
[production] |
| 12:47 |
<cdanis> |
cdanis@prometheus1003.eqiad.wmnet ~ % sudo run-puppet-agent --enable "staged rollout T222105 by cdanis" |
[production] |
| 12:41 |
<arturo> |
merging a sudo puppet module change |
[production] |
| 12:39 |
<cdanis> |
cdanis@prometheus1004.eqiad.wmnet ~ % sudo run-puppet-agent --enable "staged rollout T222105 by cdanis" |
[production] |
| 12:34 |
<elukey> |
moved /home to /srv/home (more space in a dedicated partition) on stat1005 |
[production] |
| 12:32 |
<cdanis> |
cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin 'R:prometheus::server' 'disable-puppet "staged rollout T222105 by cdanis"' |
[production] |
| 11:27 |
<Lucas_WMDE> |
EU SWAT done |
[production] |
| 11:22 |
<mlitn@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Allow cross-site requests from mobile domains (duration: 00m 52s) |
[production] |
| 11:15 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:507032|Serialize empty lists as objects on Commons (T138104)]] (duration: 00m 54s) |
[production] |
| 11:12 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:507031|Serialize empty lists as objects on Wikidata (T138104)]] (duration: 00m 55s) |
[production] |
| 11:08 |
<gilles@deploy1001> |
Finished deploy [performance/navtiming@d6756c0]: T221848 Proper fix for partitions_for_topic in python-kafka > 1.4.4 (duration: 00m 05s) |
[production] |
| 11:08 |
<gilles@deploy1001> |
Started deploy [performance/navtiming@d6756c0]: T221848 Proper fix for partitions_for_topic in python-kafka > 1.4.4 |
[production] |
| 11:02 |
<ema> |
cp3038 mbox lag, restarting varnish-be |
[production] |
| 10:55 |
<kart_> |
Updated cxserver to 2019-04-30-055331-production (T219412) |
[production] |
| 10:49 |
<santhosh@deploy1001> |
scap-helm cxserver finished |
[production] |
| 10:49 |
<santhosh@deploy1001> |
scap-helm cxserver cluster codfw completed |
[production] |
| 10:49 |
<santhosh@deploy1001> |
scap-helm cxserver upgrade -f cxserver-codfw-values.yaml production stable/cxserver [namespace: cxserver, clusters: codfw] |
[production] |
| 10:48 |
<santhosh@deploy1001> |
scap-helm cxserver finished |
[production] |
| 10:48 |
<santhosh@deploy1001> |
scap-helm cxserver cluster eqiad completed |
[production] |
| 10:48 |
<santhosh@deploy1001> |
scap-helm cxserver upgrade -f cxserver-eqiad-values.yaml production stable/cxserver [namespace: cxserver, clusters: eqiad] |
[production] |
| 10:45 |
<santhosh@deploy1001> |
scap-helm cxserver finished |
[production] |
| 10:45 |
<santhosh@deploy1001> |
scap-helm cxserver cluster staging completed |
[production] |
| 10:45 |
<santhosh@deploy1001> |
scap-helm cxserver upgrade -f cxserver-staging-values.yaml staging stable/cxserver [namespace: cxserver, clusters: staging] |
[production] |
| 10:32 |
<godog> |
rollout rsyslog upgrade to 8.1901.0-1~bpo9+wmf1 in codfw |
[production] |
| 10:32 |
<arturo> |
T222060 reimaged labtestservices2003 as stretch spare system |
[production] |
| 10:32 |
<arturo> |
T222057 reimaged labtestvirt2003 as spare system |
[production] |
| 10:12 |
<godog> |
rollout rsyslog upgrade to 8.1901.0-1~bpo9+wmf1 in eqsin / ulsfo / esams |
[production] |
| 10:08 |
<jynus> |
stop s7 and x1 instances on dbstore2* for cloning T220572 |
[production] |
| 09:31 |
<fsero@puppetmaster1001> |
conftool action : set/pooled=yes; selector: cluster=docker-registry,service=docker-registry |
[production] |