|
2019-09-20
§
|
| 09:31 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) |
[production] |
| 09:31 |
<jbond@cumin1001> |
Updating IPMI password on 1 hosts - jbond@cumin1001 |
[production] |
| 09:31 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.ipmi-password-reset |
[production] |
| 09:30 |
<jbond@cumin1001> |
END (ERROR) - Cookbook sre.hosts.ipmi-password-reset (exit_code=97) |
[production] |
| 09:30 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.ipmi-password-reset |
[production] |
| 09:30 |
<jbond@cumin1001> |
END (FAIL) - Cookbook sre.hosts.ipmi-password-reset (exit_code=99) |
[production] |
| 09:30 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.ipmi-password-reset |
[production] |
| 08:52 |
<jynus> |
creating new database on m1 "bacula9" T229209 |
[production] |
| 08:28 |
<hashar> |
Killed zuul-server process on contint2001 which was establishing connections to Gerrit and filling the pool of allowed ssh connections # T233390 |
[production] |
| 08:23 |
<hashar> |
CI in default since it is somehow no more able to fetch from Gerrit T233390 |
[production] |
| 08:20 |
<hashar> |
contint1001: upgrade zuul to 2.5.1-wmf10 # T203846 |
[production] |
| 08:12 |
<hashar> |
contint2001: upgrade zuul to 2.5.1-wmf10 # T203846 |
[production] |
| 07:46 |
<jmm@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=False) |
[production] |
| 07:46 |
<jmm@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 07:46 |
<jmm@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=False) |
[production] |
| 07:45 |
<jmm@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 07:29 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=False) |
[production] |
| 07:28 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 07:14 |
<godog> |
eqiad-prod: start ms-be1027 decom - T233289 |
[production] |
| 05:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1089 from logpager and contributions after testing, repool back with normal weight on main traffic T223151', diff saved to https://phabricator.wikimedia.org/P9136 and previous config saved to /var/cache/conftool/dbconfig/20190920-052902-marostegui.json |
[production] |
| 05:27 |
<marostegui> |
Analyze table enwiki.logging on db2102 - T223151 |
[production] |
| 05:07 |
<marostegui> |
Remove temporary index on hiwikisource views T219374 |
[production] |
| 05:07 |
<marostegui> |
Remove temporary index on hiwikisource views /usr/local/sbin/maintain-views --databases $a --replace-all --clean |
[production] |
| 01:06 |
<mholloway-shell@deploy1001> |
Finished deploy [recommendation-api/deploy@a29da76]: Rolling back deployment due to alerts beginning after 0:00 UTC (duration: 02m 51s) |
[production] |
| 01:05 |
<jforrester@deploy1001> |
Synchronized php-1.34.0-wmf.23/extensions/TimedMediaHandler/: T233360 Fix Safari 13.0 regression in video playback with audio (duration: 00m 58s) |
[production] |
| 01:03 |
<mholloway-shell@deploy1001> |
Started deploy [recommendation-api/deploy@a29da76]: Rolling back deployment due to alerts beginning after 0:00 UTC |
[production] |
|
2019-09-19
§
|
| 23:23 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 23:21 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:51 |
<ejegg> |
updated payments-wiki from adef0e858f to 939b771800 |
[production] |
| 22:34 |
<mutante> |
gerrit1001 - stopping puppet, removing gerrit IP from interface, rebooting |
[production] |
| 21:37 |
<niharika29@deploy1001> |
Synchronized wmf-config/VariantSettings.php: Enable special:mute on testwiki; T231577 (duration: 00m 56s) |
[production] |
| 20:15 |
<XioNoX> |
push firewall policies to pfw3-eqiad - T233325 |
[production] |
| 20:07 |
<XioNoX> |
push firewall policies to pfw3-codfw - T233325 |
[production] |
| 19:25 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.34.0-wmf.23 refs T220748 |
[production] |
| 19:02 |
<twentyafterfour> |
There are currently no blockers for T220748 so I am preparing to deploy 1.34.0-wmf.23 to all wikis. |
[production] |
| 18:54 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1298.eqiad.wmnet |
[production] |
| 18:14 |
<XioNoX> |
add TCP-MSS 1436 to cr2-eqiad external interfaces - T232602 |
[production] |
| 18:12 |
<XioNoX> |
add TCP-MSS 1436 to cr1-eqiad external interfaces - T232602 |
[production] |
| 18:01 |
<bblack> |
lvs2004 - restart pybal for deploy/test of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/536324/ |
[production] |
| 17:55 |
<mutante> |
puppetmaster1001 - add mcrouter cert for mw1298.eqiad.wmnet (T192457) |
[production] |
| 17:52 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'wikifeeds' for release 'staging' . |
[production] |
| 17:48 |
<krinkle@deploy1001> |
Synchronized php-1.34.0-wmf.23/extensions/AbuseFilter/includes/: T156095, 32cf50453cd (duration: 01m 04s) |
[production] |
| 17:47 |
<arlolra@deploy1001> |
Finished deploy [parsoid/deploy@77630c5]: Updating Parsoid to 6bf23c2 (duration: 08m 52s) |
[production] |
| 17:43 |
<Krinkle> |
Move whisper/MediaWiki/wanobjectcache/revision_row_1/29 to whisper/MediaWiki/wanobjectcache/revision_row_1_29 on graphite1004 and graphite2003 (T232907) |
[production] |
| 17:38 |
<arlolra@deploy1001> |
Started deploy [parsoid/deploy@77630c5]: Updating Parsoid to 6bf23c2 |
[production] |
| 17:27 |
<bblack> |
lvs2006 - restart pybal for deploy/test of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/536324/ |
[production] |
| 17:27 |
<krinkle@deploy1001> |
Synchronized php-1.34.0-wmf.23/includes/libs/objectcache/wancache: 2e910c9d3f8c04f7db, T232907 (duration: 01m 03s) |
[production] |
| 17:23 |
<bblack> |
lvs2005 - restart pybal for deploy/test of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/536324/ |
[production] |
| 17:19 |
<bblack> |
lvs2006 - restart pybal for deploy/test of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/536324/ |
[production] |
| 17:16 |
<bblack> |
lvs200[456] - puppet disabled for https://gerrit.wikimedia.org/r/536324 deploy/test |
[production] |