4651-4700 of 10000 results (56ms)
2019-04-30 ยง
16:57 <ayounsi@deploy1001> Finished deploy [librenms/librenms@0fd8da6]: Rollback LibreNMS (duration: 00m 09s) [production]
16:57 <ayounsi@deploy1001> Started deploy [librenms/librenms@0fd8da6]: Rollback LibreNMS [production]
16:52 <arturo> merging change to `profile::base` and `::raid` https://gerrit.wikimedia.org/r/c/operations/puppet/+/507357 related to T221225 [production]
16:36 <ayounsi@deploy1001> Finished deploy [librenms/librenms@2094575]: Upgrade LibreNMS to 1.51 - T207706 (duration: 00m 11s) [production]
16:36 <ayounsi@deploy1001> Started deploy [librenms/librenms@2094575]: Upgrade LibreNMS to 1.51 - T207706 [production]
16:27 <XioNoX> upgrade librenms to 1.51 [production]
16:26 <jbond42> upgrade puppet and facter in eqsin [production]
16:04 <ema> pool cp4022 w/ ATS backend T219967 [production]
15:58 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
15:58 <robh@cumin1001> START - Cookbook sre.hosts.decommission [production]
15:58 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
15:58 <robh@cumin1001> START - Cookbook sre.hosts.decommission [production]
15:45 <elukey> restart hadoop hdfs namenodes on an-master100[1,2] to pick up new logging settings - T220702 [production]
15:18 <jynus> stop s8 instance on dbstore2001 for cloning to db2100 T220572 [production]
15:09 <jiji@deploy1001> Synchronized wmf-config/CommonSettings.php: Send 1% of anonymous users to PHP7.2 - T219150 (duration: 00m 54s) [production]
14:58 <jbond42> enable-puppet "T220987: global kafaka log shipping - staged rollout (jbond)" [production]
14:56 <cdanis> cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin 'bast3002*' 'run-puppet-agent --enable "filippo prometheus"' [production]
14:49 <cdanis> cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin 'labmon1001*' 'run-puppet-agent --enable "staged rollout T222105 by cdanis"' [production]
14:44 <jijiki> Sending 1% of anonymous users to PHP7.2 - T219150 [production]
14:43 <cdanis> cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin 'bast5001*' 'run-puppet-agent --enable "staged rollout T222105 by cdanis"' [production]
14:26 <jbond42> disable-puppet "T220987: global kafaka log shipping - staged rollout (jbond)" [production]
14:24 <cdanis> cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin 'prometheus2004*' 'run-puppet-agent --enable "staged rollout T222105 by cdanis"' [production]
14:17 <cdanis> cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin 'prometheus2003*' 'run-puppet-agent --enable "staged rollout T222105 by cdanis"' [production]
14:15 <cdanis> cdanis@prometheus1003.eqiad.wmnet ~ % sudo enable-puppet 'cdanis testing original query.max-samples T222105' [production]
13:29 <cdanis> cdanis@prometheus1004.eqiad.wmnet ~ % sudo systemctl restart prometheus@ops.service [production]
13:28 <ema> depool cp4022 and reimage as upload_ats T219967 [production]
13:20 <arturo> reverting sudo puppet module changes https://gerrit.wikimedia.org/r/c/operations/puppet/+/507317 [production]
13:16 <cdanis> cdanis@prometheus1003.eqiad.wmnet ~ % sudo systemctl restart prometheus@ops.service [production]
13:15 <cdanis> cdanis@prometheus1003.eqiad.wmnet ~ % sudo disable-puppet 'cdanis testing original query.max-samples T222105' [production]
13:08 <cdanis> OOMed the eqiad ops prometheus @ prometheus1003 [production]
13:02 <cdanis> OOMed the eqiad ops prometheus @ prometheus1004 [production]
12:47 <cdanis> cdanis@prometheus1003.eqiad.wmnet ~ % sudo run-puppet-agent --enable "staged rollout T222105 by cdanis" [production]
12:41 <arturo> merging a sudo puppet module change [production]
12:39 <cdanis> cdanis@prometheus1004.eqiad.wmnet ~ % sudo run-puppet-agent --enable "staged rollout T222105 by cdanis" [production]
12:34 <elukey> moved /home to /srv/home (more space in a dedicated partition) on stat1005 [production]
12:32 <cdanis> cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin 'R:prometheus::server' 'disable-puppet "staged rollout T222105 by cdanis"' [production]
11:27 <Lucas_WMDE> EU SWAT done [production]
11:22 <mlitn@deploy1001> Synchronized wmf-config/CommonSettings.php: Allow cross-site requests from mobile domains (duration: 00m 52s) [production]
11:15 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:507032|Serialize empty lists as objects on Commons (T138104)]] (duration: 00m 54s) [production]
11:12 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:507031|Serialize empty lists as objects on Wikidata (T138104)]] (duration: 00m 55s) [production]
11:08 <gilles@deploy1001> Finished deploy [performance/navtiming@d6756c0]: T221848 Proper fix for partitions_for_topic in python-kafka > 1.4.4 (duration: 00m 05s) [production]
11:08 <gilles@deploy1001> Started deploy [performance/navtiming@d6756c0]: T221848 Proper fix for partitions_for_topic in python-kafka > 1.4.4 [production]
11:02 <ema> cp3038 mbox lag, restarting varnish-be [production]
10:55 <kart_> Updated cxserver to 2019-04-30-055331-production (T219412) [production]
10:49 <santhosh@deploy1001> scap-helm cxserver finished [production]
10:49 <santhosh@deploy1001> scap-helm cxserver cluster codfw completed [production]
10:49 <santhosh@deploy1001> scap-helm cxserver upgrade -f cxserver-codfw-values.yaml production stable/cxserver [namespace: cxserver, clusters: codfw] [production]
10:48 <santhosh@deploy1001> scap-helm cxserver finished [production]
10:48 <santhosh@deploy1001> scap-helm cxserver cluster eqiad completed [production]
10:48 <santhosh@deploy1001> scap-helm cxserver upgrade -f cxserver-eqiad-values.yaml production stable/cxserver [namespace: cxserver, clusters: eqiad] [production]