|
2017-05-02
§
|
| 10:11 |
<jynus> |
stopping replication on db1015 |
[production] |
| 09:58 |
<END> |
(PASS) - Resync the redis for jobqueues in eqiad with the masters in codfw - t04_resync_redis (switchdc/oblivian@neodymium) |
[production] |
| 09:56 |
<START> |
- Resync the redis for jobqueues in eqiad with the masters in codfw - t04_resync_redis (switchdc/oblivian@neodymium) |
[production] |
| 09:55 |
<_joe_> |
testing pre-switchover the step to restart & resync redises in dc_to (eqiad) |
[production] |
| 09:53 |
<joal> |
Restart mediawiki history jobs to pick up new snapshot format |
[analytics] |
| 09:48 |
<jynus@naos> |
Synchronized wmf-config/db-codfw.php: Add db1097 (duration: 01m 00s) |
[production] |
| 09:47 |
<jynus@naos> |
Synchronized wmf-config/db-eqiad.php: Depool db1015 & add db1097 (duration: 01m 17s) |
[production] |
| 09:41 |
<godog> |
flip deployment-cache-upload04 to deployment-ms-fe02 - T162247 |
[releng] |
| 09:36 |
<hashar> |
Jenkins/CI is back up! |
[production] |
| 09:34 |
<hashar> |
Nodepool can not add instances to Jenkins any more. Roll backing Jenkins to 2.32.3 |
[production] |
| 09:29 |
<akosiaris> |
Set description for ganeti2005, ganeti2006 on asw-a-codfw. T164011 |
[production] |
| 09:27 |
<akosiaris> |
create interface range ganeti on asw-a-codfw. T164011 |
[production] |
| 09:24 |
<akosiaris> |
remove configuration from ge-8/0/0, ge-8/0/3 from asw-b-codfw for ganeti2005, ganeti2006 move to row A. T164011 |
[production] |
| 09:21 |
<hashar> |
Starting Nodepool |
[production] |
| 09:16 |
<hashar> |
Stopping Nodepool |
[production] |
| 09:14 |
<hashar> |
OpenStack / wmflabs fails to create new instances |
[production] |
| 08:40 |
<hashar> |
Upgrading Jenkins to 2.46.2 - T144106 |
[production] |
| 08:40 |
<elukey> |
run puppet and restart nutcracker on eqiad hosts with profile::mediawiki::nutcracker |
[production] |
| 08:33 |
<hashar> |
Upgrading Jenkins to 2.32.3 - T144106 |
[production] |
| 08:32 |
<elukey> |
stop and mask redis on mc1001-mc1018 - T137345 |
[production] |
| 08:26 |
<hashar> |
Upgrading Jenkins to 2.19.4 - T144106 |
[production] |
| 08:17 |
<hashar> |
Reconfigured all Jenkins jobs via jjb |
[releng] |
| 08:14 |
<hashar> |
Installing Jenkins Pipeline plugin |
[production] |
| 08:04 |
<hashar> |
Installing Jenkins plugin Pipeline: Stage View https://plugins.jenkins.io/pipeline-stage-view |
[production] |
| 08:04 |
<hashar> |
Upgrading Jenkins to 2.7.4 - T144106 |
[production] |
| 07:59 |
<elukey> |
Swap mc1001->mc1012 with mc1019->mc2030 - T137345 (more informative :) |
[production] |
| 07:58 |
<elukey> |
wap mc1001->mc1012 with mc1019->mc2030 |
[production] |
| 07:36 |
<_joe_> |
starting etcd replication codfw => eqiad |
[production] |
| 06:46 |
<_joe_> |
disabling etcd auth on conf1*, converting to use nginx for TLS/auth T159687 |
[production] |
| 05:21 |
<bd808> |
Deleted $HOME/service.manifest; webservice restart loop gone crazy |
[tools.webarchivebot] |
| 03:43 |
<bd808> |
Setup striker-puppet01.striker.eqiad.wmflabs as project puppetmaster |
[striker] |
| 03:10 |
<mattflaschen@naos> |
Synchronized php-1.29.0-wmf.21/extensions/FlaggedRevs/: Urgent deploy: Fix FlaggedRevs fatal, and also a filter issue: T164096 and T164049 (duration: 00m 56s) |
[production] |
| 02:45 |
<tstarling@naos> |
Synchronized php-1.29.0-wmf.21/includes/config/EtcdConfig.php: EtcdConfig backported bug fixes (duration: 01m 02s) |
[production] |
| 02:34 |
<tstarling@naos> |
Synchronized wmf-config/CommonSettings.php: siteinfo hook (duration: 02m 39s) |
[production] |
| 00:33 |
<tstarling@puppetmaster1001> |
conftool action : set/@read-write.yaml; selector: name=ReadOnly |
[production] |
| 00:33 |
<tstarling@puppetmaster1001> |
conftool action : set/@dc-codfw.yaml; selector: name=WMFMasterDatacenter |
[production] |
| 00:25 |
<TimStarling> |
populating production etcd with initial mediawiki config keys |
[production] |
|
2017-05-01
§
|
| 23:41 |
<mutante> |
netmon1002 - signed puppet cert, initial puppet run, accept salt-key,.. (T159756) |
[production] |
| 23:32 |
<bd808> |
Reverting to labs puppetmaster actually worked, but now I remember why there was a project puppetmaster :/ |
[striker] |
| 23:15 |
<mutante> |
netmon1002 - boot into PXE, initial OS install (T159756) |
[production] |
| 23:06 |
<bd808> |
Ran puppet cert clean striker-deploy03.striker.eqiad.wmflabs on labcontrol1001 |
[production] |
| 22:58 |
<bd808> |
Trying to revert from role::puppet::self to labs puppetmaster |
[striker] |
| 21:21 |
<bd808> |
Deleting mw-vagrant managed VM on striker-deploy03 to rebuild it as a jessie VM |
[striker] |
| 20:39 |
<hashar> |
Updated REL1_29 branch of ImportArticles / OAuth / Quiz and Wikispeech so they get phpcs ( https://gerrit.wikimedia.org/r/#/c/350984/ ) |
[releng] |
| 20:26 |
<hashar> |
nodepool: deleting alien instance ci-trusty-wikimedia-631443 4e66ad7e-b9d3-4af1-b559-3f54968d376e |
[releng] |
| 19:43 |
<ejegg> |
updated payments-wiki from 4c5630283c57efbc454cc70d47218f7f22ea252a to 57451dee67e498d445a6f9bc10d40acf3df65f38 |
[production] |
| 19:10 |
<mobrovac@naos> |
Finished deploy [mobileapps/deploy@b5afcb8]: Forced deploy to bring the targets to the current version (duration: 02m 08s) |
[production] |
| 19:08 |
<mobrovac@naos> |
Started deploy [mobileapps/deploy@b5afcb8]: Forced deploy to bring the targets to the current version |
[production] |
| 18:46 |
<mutante> |
temp. re-enabling puppet on restbase1018 and running it once to fix icinga config syntax error. then disabling it again. restbase service stopped before and after. this box has a broken disk. |
[production] |
| 18:35 |
<mutante> |
brought mc1018 back up, ran puppet on it and then on Icinga. parent was adjusted from asw-d-eqiad to asw2-2-eqiad. reduced icinga config errors by 50% :p (1 of 2 left, restbase1018) |
[production] |