|
2021-11-30
§
|
| 13:25 |
<elukey@cumin1001> |
START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons for openjdk upgrade. |
[production] |
| 13:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'After maintenance db2114 (T277354)', diff saved to https://phabricator.wikimedia.org/P17904 and previous config saved to /var/cache/conftool/dbconfig/20211130-131124-marostegui.json |
[production] |
| 13:05 |
<topranks> |
Running homer against CR routers to adjust loopback4 filter enabling local NTP queries for status. T296623 |
[production] |
| 12:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'After maintenance db2114 (T277354)', diff saved to https://phabricator.wikimedia.org/P17903 and previous config saved to /var/cache/conftool/dbconfig/20211130-125620-marostegui.json |
[production] |
| 12:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'After maintenance db2114 (T277354)', diff saved to https://phabricator.wikimedia.org/P17902 and previous config saved to /var/cache/conftool/dbconfig/20211130-124115-marostegui.json |
[production] |
| 12:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'After maintenance db2114 (T277354)', diff saved to https://phabricator.wikimedia.org/P17901 and previous config saved to /var/cache/conftool/dbconfig/20211130-122610-marostegui.json |
[production] |
| 12:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2114 (T277354)', diff saved to https://phabricator.wikimedia.org/P17900 and previous config saved to /var/cache/conftool/dbconfig/20211130-122555-marostegui.json |
[production] |
| 12:25 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2114.codfw.wmnet with reason: Maintenance T277354 |
[production] |
| 12:25 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2114.codfw.wmnet with reason: Maintenance T277354 |
[production] |
| 12:09 |
<jbond@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts puppetboard1001.eqiad.wmnet |
[production] |
| 12:02 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts puppetboard1001.eqiad.wmnet |
[production] |
| 11:50 |
<moritzm> |
running "sudo gnt-cluster renew-crypto --new-node-certificates --new-rapi-certificate --new-spice-certificate" for Ganeti codfw cluster T296622 |
[production] |
| 11:01 |
<hnowlan> |
restarting tilerator, kartotherian and tileratorui for updates in eqiad |
[production] |
| 11:01 |
<hnowlan> |
restarting tilerator, kartotherian and tileratorui in codfw |
[production] |
| 10:39 |
<elukey> |
rollout wmf-certificates 0~20211129-1 fleet wide (add group/others permissions to the cert bundle) |
[production] |
| 10:30 |
<lucaswerkmeister-wmde@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'termbox' for release 'production' . |
[production] |
| 10:29 |
<lucaswerkmeister-wmde@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'termbox' for release 'production' . |
[production] |
| 09:58 |
<moritzm> |
installing remaining ICU security updates |
[production] |
| 09:06 |
<Amir1> |
dropping wikiadmin@localhost from all pooled replicas of s6 (T296511) |
[production] |
| 08:24 |
<dcausse> |
restarting blazegraph on wdqs1006 (jvm stuck for 6hours) |
[production] |
| 08:14 |
<Amir1> |
revoking DROP from wikiadmin on all pooled replicas (T249683) |
[production] |
| 03:46 |
<ejegg> |
updated payments-wiki from dbc92132 to 4a4ef51d |
[production] |
| 02:05 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 02:04 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:22 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:20 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:17 |
<catrope@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:742524|Enable scroll tracking for all users (T292586)]] (duration: 00m 55s) |
[production] |
| 00:14 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:14 |
<catrope@deploy1002> |
Synchronized php-1.38.0-wmf.9/extensions/WikimediaEvents/modules/ext.wikimediaEvents/readingDepth.js: Backport: [[gerrit:742517|Provide fallback for config variable when not present]] (duration: 00m 55s) |
[production] |
| 00:13 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:13 |
<catrope@deploy1002> |
Synchronized wmf-config/flaggedrevs.php: Config: [[gerrit:738530|allow sysops to set/remove reviewer group on ckbwiki (T294696)]] (duration: 00m 55s) |
[production] |
|
2021-11-29
§
|
| 22:32 |
<sbassett@deploy1002> |
Synchronized php-1.38.0-wmf.9/extensions/EntitySchema/src/MediaWiki/Specials/SetEntitySchemaLabelDescriptionAliases.php: Deploy security patch for T296578 (duration: 00m 55s) |
[production] |
| 22:21 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 22:20 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 22:20 |
<sbassett@deploy1002> |
Synchronized php-1.38.0-wmf.9/extensions/FileImporter/src/Remote/MediaWiki/HttpApiLookup.php: Backport: [[gerrit:742263|SECURITY: Fix special page displaying unescaped user input (T296605)]] (duration: 00m 56s) |
[production] |
| 20:48 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:47 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:46 |
<jforrester@deploy1002> |
Synchronized wmf-config/CommonSettings.php: Fix wgWikiLambdaOrchestratorLocation service pointer typo (duration: 00m 55s) |
[production] |
| 20:27 |
<tgr> |
UTC evening deploys done |
[production] |
| 20:26 |
<tgr@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:742261|GrowthExperiments: Start imagerecommendation variant experiment]] (duration: 00m 55s) |
[production] |
| 20:23 |
<tgr@deploy1002> |
Synchronized php-1.38.0-wmf.9/extensions/GrowthExperiments/includes/NewcomerTasks/AddImage/AddImageSubmissionHandler.php: Backport: [[gerrit:742262|AddImage: Refresh user's task feed after undecided rejection (T296491)]] (duration: 00m 56s) |
[production] |
| 20:21 |
<tgr@deploy1002> |
Synchronized php-1.38.0-wmf.9/extensions/GrowthExperiments/includes/HomepageModules/SuggestedEdits.php: Backport: [[gerrit:742260|SuggestedEdits: Drop isActivated() check in getJsData (T296626)]] (duration: 00m 56s) |
[production] |
| 20:17 |
<ejegg> |
updated payments-wiki from d1d6f024 -> dbc92132 |
[production] |
| 20:16 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:15 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:10 |
<eileen> |
civicrm |
[production] |
| 20:09 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:08 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:02 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:01 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |