|
2019-04-30
ยง
|
| 13:20 |
<arturo> |
reverting sudo puppet module changes https://gerrit.wikimedia.org/r/c/operations/puppet/+/507317 |
[production] |
| 13:16 |
<cdanis> |
cdanis@prometheus1003.eqiad.wmnet ~ % sudo systemctl restart prometheus@ops.service |
[production] |
| 13:15 |
<cdanis> |
cdanis@prometheus1003.eqiad.wmnet ~ % sudo disable-puppet 'cdanis testing original query.max-samples T222105' |
[production] |
| 13:08 |
<cdanis> |
OOMed the eqiad ops prometheus @ prometheus1003 |
[production] |
| 13:02 |
<cdanis> |
OOMed the eqiad ops prometheus @ prometheus1004 |
[production] |
| 12:47 |
<cdanis> |
cdanis@prometheus1003.eqiad.wmnet ~ % sudo run-puppet-agent --enable "staged rollout T222105 by cdanis" |
[production] |
| 12:41 |
<arturo> |
merging a sudo puppet module change |
[production] |
| 12:39 |
<cdanis> |
cdanis@prometheus1004.eqiad.wmnet ~ % sudo run-puppet-agent --enable "staged rollout T222105 by cdanis" |
[production] |
| 12:34 |
<elukey> |
moved /home to /srv/home (more space in a dedicated partition) on stat1005 |
[production] |
| 12:32 |
<cdanis> |
cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin 'R:prometheus::server' 'disable-puppet "staged rollout T222105 by cdanis"' |
[production] |
| 11:27 |
<Lucas_WMDE> |
EU SWAT done |
[production] |
| 11:22 |
<mlitn@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Allow cross-site requests from mobile domains (duration: 00m 52s) |
[production] |
| 11:15 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:507032|Serialize empty lists as objects on Commons (T138104)]] (duration: 00m 54s) |
[production] |
| 11:12 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:507031|Serialize empty lists as objects on Wikidata (T138104)]] (duration: 00m 55s) |
[production] |
| 11:08 |
<gilles@deploy1001> |
Finished deploy [performance/navtiming@d6756c0]: T221848 Proper fix for partitions_for_topic in python-kafka > 1.4.4 (duration: 00m 05s) |
[production] |
| 11:08 |
<gilles@deploy1001> |
Started deploy [performance/navtiming@d6756c0]: T221848 Proper fix for partitions_for_topic in python-kafka > 1.4.4 |
[production] |
| 11:02 |
<ema> |
cp3038 mbox lag, restarting varnish-be |
[production] |
| 10:55 |
<kart_> |
Updated cxserver to 2019-04-30-055331-production (T219412) |
[production] |
| 10:49 |
<santhosh@deploy1001> |
scap-helm cxserver finished |
[production] |
| 10:49 |
<santhosh@deploy1001> |
scap-helm cxserver cluster codfw completed |
[production] |
| 10:49 |
<santhosh@deploy1001> |
scap-helm cxserver upgrade -f cxserver-codfw-values.yaml production stable/cxserver [namespace: cxserver, clusters: codfw] |
[production] |
| 10:48 |
<santhosh@deploy1001> |
scap-helm cxserver finished |
[production] |
| 10:48 |
<santhosh@deploy1001> |
scap-helm cxserver cluster eqiad completed |
[production] |
| 10:48 |
<santhosh@deploy1001> |
scap-helm cxserver upgrade -f cxserver-eqiad-values.yaml production stable/cxserver [namespace: cxserver, clusters: eqiad] |
[production] |
| 10:45 |
<santhosh@deploy1001> |
scap-helm cxserver finished |
[production] |
| 10:45 |
<santhosh@deploy1001> |
scap-helm cxserver cluster staging completed |
[production] |
| 10:45 |
<santhosh@deploy1001> |
scap-helm cxserver upgrade -f cxserver-staging-values.yaml staging stable/cxserver [namespace: cxserver, clusters: staging] |
[production] |
| 10:32 |
<godog> |
rollout rsyslog upgrade to 8.1901.0-1~bpo9+wmf1 in codfw |
[production] |
| 10:32 |
<arturo> |
T222060 reimaged labtestservices2003 as stretch spare system |
[production] |
| 10:32 |
<arturo> |
T222057 reimaged labtestvirt2003 as spare system |
[production] |
| 10:12 |
<godog> |
rollout rsyslog upgrade to 8.1901.0-1~bpo9+wmf1 in eqsin / ulsfo / esams |
[production] |
| 10:08 |
<jynus> |
stop s7 and x1 instances on dbstore2* for cloning T220572 |
[production] |
| 09:31 |
<fsero@puppetmaster1001> |
conftool action : set/pooled=yes; selector: cluster=docker-registry,service=docker-registry |
[production] |
| 09:26 |
<fsero> |
creating lvs endpoints for docker registry - T221101 |
[production] |
| 09:02 |
<elukey> |
roll restart hdfs namenodes on an-master100[1,2] to pick up new settings - T220702 |
[production] |
| 08:22 |
<godog> |
bounce prometheus on bast4002 after backfill has finished - T187987 |
[production] |
| 08:11 |
<gilles@deploy1001> |
Finished deploy [performance/navtiming@8f135ac]: T221848 Default to partition 0 when no partition is found (duration: 00m 05s) |
[production] |
| 08:11 |
<gilles@deploy1001> |
Started deploy [performance/navtiming@8f135ac]: T221848 Default to partition 0 when no partition is found |
[production] |
| 08:11 |
<gilles@deploy1001> |
deploy aborted: T221848 Defalt to partition 0 when no partition is found (duration: 00m 00s) |
[production] |
| 08:11 |
<gilles@deploy1001> |
Started deploy [performance/navtiming@8f135ac]: T221848 Defalt to partition 0 when no partition is found |
[production] |
| 07:53 |
<gilles@deploy1001> |
Finished deploy [performance/navtiming@e900152]: T221848 add more logging around startup (duration: 00m 05s) |
[production] |
| 07:53 |
<gilles@deploy1001> |
Started deploy [performance/navtiming@e900152]: T221848 add more logging around startup |
[production] |
| 07:29 |
<moritzm> |
installing systemd updates for jessie |
[production] |
| 07:24 |
<marostegui> |
Remove labservices1001 and labservices1002 from tendril T221857 |
[production] |
| 05:27 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Clarify db1093's status (duration: 00m 51s) |
[production] |
| 05:26 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Clarify db1093's status (duration: 00m 55s) |
[production] |
| 04:26 |
<mutante> |
LDAP - remove user pirroh from group nda (T222085 and cross-validate-accounts demands consistency) |
[production] |
| 02:23 |
<mutante> |
analytics1050 - systemctl start mclog ... it was failed like recently on analytics1052 (T212219 ?) |
[production] |
| 02:09 |
<tgr@deploy1001> |
Synchronized wmf-config/db-eqiad.php: SWAT: [[gerrit:507237|depool db1093]] (duration: 00m 54s) |
[production] |
| 01:30 |
<mutante> |
contint2001..then contint1001 - deleting /etc/zuul/wikimedia and letting puppet re-clone it (gerrit:507070) (T218844) |
[production] |