|
2020-10-05
§
|
| 23:11 |
<ejegg> |
updated payments staging from 52704ffe24 to db03677b2d |
[production] |
| 22:31 |
<Amir1> |
deleted deployment-mailman01 (T257118) |
[releng] |
| 22:29 |
<Amir1> |
deleted deployment-imagescaler01 and deployment-imagescaler02 (T257118) |
[releng] |
| 22:29 |
<mutante> |
deleted the shinken module |
[monitoring] |
| 22:27 |
<mutante> |
removing shinken puppet module and role |
[production] |
| 22:01 |
<ebernhardson> |
restore wikidatawiki_content enwiki_content enwiki_general and commonswiki_file to default index.merge.policy.deletes_pct_allowed on eqiad cirrus cluster T264053 |
[production] |
| 21:58 |
<bstorm> |
setting "mtail::from_component: true" on both mx-out servers to make puppet work again |
[cloudinfra] |
| 21:01 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 21:00 |
<Urbanecm> |
Run keyholder arm at deployment-cumin |
[releng] |
| 21:00 |
<Urbanecm> |
Run puppet at deployment-mwmaint01 and deployment-mediawiki-07 |
[releng] |
| 20:59 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:45 |
<hauskatze> |
Fixed puppet for deployment-mediawiki-07: s/memcache/memcached/ |
[releng] |
| 20:30 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 20:28 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:26 |
<ebernhardson> |
restart elasticsearch_6@production-search-codfw on elastic2051 to take reduced (32 sector, 16kB) readahead settings T264053 |
[production] |
| 20:13 |
<ebernhardson> |
restart elasticsearch_6@production-search-codfw on elastic2051 to take reduced (64 sector, 32kB) readahead settings T264053 |
[production] |
| 19:56 |
<ebernhardson> |
restart elasticsearch_6@production-search-codfw on elastic2050 to take reduced (128kB) readahead settings T264053 |
[production] |
| 19:31 |
<mutante> |
ran sre.dns.netbox to push addition of an-worker1113 which was commited in prod repo but not in netbox data |
[production] |
| 19:30 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 19:27 |
<dzahn@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 19:14 |
<mforns> |
restarted oozie coord unique_devices-per_domain-monthly after deployment |
[analytics] |
| 19:05 |
<mforns> |
finished deploying refinery to unblock deletion of raw mediawiki_job and raw netflow data |
[analytics] |
| 18:59 |
<mforns@deploy1001> |
Finished deploy [analytics/refinery@2c6c335] (thin): [THIN] Special deployment to unblock deletion jobs [analytics/refinery@2c6c335e61cecd0321ec6f066a153feaf2dbbc27] (duration: 00m 08s) |
[production] |
| 18:59 |
<mforns@deploy1001> |
Started deploy [analytics/refinery@2c6c335] (thin): [THIN] Special deployment to unblock deletion jobs [analytics/refinery@2c6c335e61cecd0321ec6f066a153feaf2dbbc27] |
[production] |
| 18:58 |
<mforns@deploy1001> |
Finished deploy [analytics/refinery@2c6c335]: Special deployment to unblock deletion jobs [analytics/refinery@2c6c335e61cecd0321ec6f066a153feaf2dbbc27] (duration: 12m 08s) |
[production] |
| 18:46 |
<mforns@deploy1001> |
Started deploy [analytics/refinery@2c6c335]: Special deployment to unblock deletion jobs [analytics/refinery@2c6c335e61cecd0321ec6f066a153feaf2dbbc27] |
[production] |
| 18:46 |
<mutante> |
marked project for deletion in 2020 purge |
[planet] |
| 18:45 |
<mforns> |
deploying refinery to unblock deletion of raw mediawiki_job and raw netflow data |
[analytics] |
| 18:44 |
<mutante> |
shutting down instance pk8s - not in use since 2019 |
[planet] |
| 18:20 |
<elukey> |
manual creation of /opt/rocm -> /opt/rocm-3.3.0 on stat1008 to avoid failures in finding the lib dir |
[analytics] |
| 18:17 |
<elukey@cumin1001> |
END (FAIL) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=99) |
[production] |
| 18:17 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.init-hadoop-workers |
[production] |
| 18:15 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) |
[production] |
| 18:13 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.init-hadoop-workers |
[production] |
| 18:11 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) |
[production] |
| 18:10 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.init-hadoop-workers |
[production] |
| 17:53 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:51 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:44 |
<wm-bot> |
<bd808> Purged cache of puppet roles for T264649 |
[tools.openstack-browser] |
| 17:40 |
<bd808> |
`service uwsgi-labspuppetbackend restart` on cloud-puppetmaster-03 (T264649) |
[admin] |
| 17:29 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:27 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:25 |
<hnowlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . |
[production] |
| 17:25 |
<hnowlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . |
[production] |
| 17:11 |
<elukey> |
bootstrap an-worker[1115-1117] as hadoop workers |
[analytics] |
| 17:00 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . |
[production] |