|
2021-11-30
ยง
|
| 16:23 |
<XioNoX> |
Move cr2-codfw pfw3 link to BO cable - T289241 |
[production] |
| 16:20 |
<Emperor> |
reboot ms-be2059 to fix device enumeration order re T295563 |
[production] |
| 16:14 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Depool db1163 at 25%', diff saved to https://phabricator.wikimedia.org/P17906 and previous config saved to /var/cache/conftool/dbconfig/20211130-161457-jynus.json |
[production] |
| 16:13 |
<XioNoX> |
cr2-codfw bounce fpc 1 pic 0 (vrrp backup) - T289241 |
[production] |
| 16:07 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Depool db1163 at 50%', diff saved to https://phabricator.wikimedia.org/P17905 and previous config saved to /var/cache/conftool/dbconfig/20211130-160748-jynus.json |
[production] |
| 16:06 |
<bblack> |
lvs2007 - repooling into service |
[production] |
| 16:01 |
<bblack> |
lvs2007 - depooling for network maint - do not push LVS config changes please! |
[production] |
| 15:41 |
<jbond@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts puppetboard2001.codfw.wmnet |
[production] |
| 15:41 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts puppetboard2001.codfw.wmnet |
[production] |
| 15:38 |
<jbond@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts puppetboard2001.codfw.wmnet |
[production] |
| 15:37 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts puppetboard2001.codfw.wmnet |
[production] |
| 15:32 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 15:29 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 15:23 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 15:22 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 15:16 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 15:15 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 15:12 |
<jforrester@deploy1002> |
Synchronized multiversion/MWMultiVersion.php: Add wikifunctions hard-coded value to setSiteInfoForWiki for Beta Cluster T284162 (duration: 00m 56s) |
[production] |
| 15:09 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 15:08 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 13:45 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons for openjdk upgrade. |
[production] |
| 13:25 |
<elukey@cumin1001> |
START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons for openjdk upgrade. |
[production] |
| 13:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'After maintenance db2114 (T277354)', diff saved to https://phabricator.wikimedia.org/P17904 and previous config saved to /var/cache/conftool/dbconfig/20211130-131124-marostegui.json |
[production] |
| 13:05 |
<topranks> |
Running homer against CR routers to adjust loopback4 filter enabling local NTP queries for status. T296623 |
[production] |
| 12:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'After maintenance db2114 (T277354)', diff saved to https://phabricator.wikimedia.org/P17903 and previous config saved to /var/cache/conftool/dbconfig/20211130-125620-marostegui.json |
[production] |
| 12:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'After maintenance db2114 (T277354)', diff saved to https://phabricator.wikimedia.org/P17902 and previous config saved to /var/cache/conftool/dbconfig/20211130-124115-marostegui.json |
[production] |
| 12:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'After maintenance db2114 (T277354)', diff saved to https://phabricator.wikimedia.org/P17901 and previous config saved to /var/cache/conftool/dbconfig/20211130-122610-marostegui.json |
[production] |
| 12:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2114 (T277354)', diff saved to https://phabricator.wikimedia.org/P17900 and previous config saved to /var/cache/conftool/dbconfig/20211130-122555-marostegui.json |
[production] |
| 12:25 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2114.codfw.wmnet with reason: Maintenance T277354 |
[production] |
| 12:25 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2114.codfw.wmnet with reason: Maintenance T277354 |
[production] |
| 12:09 |
<jbond@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts puppetboard1001.eqiad.wmnet |
[production] |
| 12:02 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts puppetboard1001.eqiad.wmnet |
[production] |
| 11:50 |
<moritzm> |
running "sudo gnt-cluster renew-crypto --new-node-certificates --new-rapi-certificate --new-spice-certificate" for Ganeti codfw cluster T296622 |
[production] |
| 11:01 |
<hnowlan> |
restarting tilerator, kartotherian and tileratorui for updates in eqiad |
[production] |
| 11:01 |
<hnowlan> |
restarting tilerator, kartotherian and tileratorui in codfw |
[production] |
| 10:39 |
<elukey> |
rollout wmf-certificates 0~20211129-1 fleet wide (add group/others permissions to the cert bundle) |
[production] |
| 10:30 |
<lucaswerkmeister-wmde@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'termbox' for release 'production' . |
[production] |
| 10:29 |
<lucaswerkmeister-wmde@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'termbox' for release 'production' . |
[production] |
| 09:58 |
<moritzm> |
installing remaining ICU security updates |
[production] |
| 09:06 |
<Amir1> |
dropping wikiadmin@localhost from all pooled replicas of s6 (T296511) |
[production] |
| 08:24 |
<dcausse> |
restarting blazegraph on wdqs1006 (jvm stuck for 6hours) |
[production] |
| 08:14 |
<Amir1> |
revoking DROP from wikiadmin on all pooled replicas (T249683) |
[production] |
| 03:46 |
<ejegg> |
updated payments-wiki from dbc92132 to 4a4ef51d |
[production] |
| 02:05 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 02:04 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:22 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:20 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:17 |
<catrope@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:742524|Enable scroll tracking for all users (T292586)]] (duration: 00m 55s) |
[production] |
| 00:14 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:14 |
<catrope@deploy1002> |
Synchronized php-1.38.0-wmf.9/extensions/WikimediaEvents/modules/ext.wikimediaEvents/readingDepth.js: Backport: [[gerrit:742517|Provide fallback for config variable when not present]] (duration: 00m 55s) |
[production] |