|
2023-04-22
§
|
| 19:35 |
<Krinkle> |
Create excimer_ui_server db user in Beta Cluster on deployment-db13 based on prod grants. Password stored in deployment-puppetmaster04:/var/lib/git/labs/private/ (passwords::excimer_ui_server::$excimer_db_pass). Picked db13 because recommendationapi is here (also prod m2), and because mdb (created for this purpose originally) appears broken since several OS iterations (likely forgotten due to unusual name) T301637, T331956 |
[releng] |
| 18:05 |
<Krinkle> |
Fix database hostname dns error at https://performance.wikimedia.beta.wmflabs.org/xhgui/, switch from mdb01 to mdb02, ref T301637 |
[releng] |
| 18:00 |
<Krinkle> |
Move deployment-xhgui03 config from to deployment-xhgui prefix, https://gerrit.wikimedia.org/g/cloud/instance-puppet/+/1d03d4cf84a148bff7e055d9939c44f30c618d85/deployment-prep/deployment-xhgui03.deployment-prep.eqiad1.wikimedia.cloud.yaml https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/ab1c9d2968ba3c292b81ba76f8c0775151630f94%5E%21/, ref T301637 |
[releng] |
| 16:18 |
<wm-bot> |
<lucaswerkmeister> deployed a074fd9c64 (trim spaces) |
[tools.lexeme-forms] |
| 15:39 |
<wm-bot> |
<lucaswerkmeister> deployed fdb0552957 (remove spaces) |
[tools.lexeme-forms] |
| 15:23 |
<wm-bot> |
<lucaswerkmeister> Double IRC messages to other bridges |
[tools.bridgebot] |
| 15:21 |
<wm-bot> |
<lucaswerkmeister> deployed b6a1268b21 (Punjabi nouns) |
[tools.lexeme-forms] |
| 05:41 |
<joe> |
<thumbor/codfw>$ helmfile --state-values-set roll_restart=1 -e codfw sync |
[production] |
| 05:40 |
<oblivian@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/thumbor: sync |
[production] |
| 05:39 |
<oblivian@deploy2002> |
helmfile [codfw] START helmfile.d/services/thumbor: sync |
[production] |
| 05:39 |
<oblivian@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/thumbor: apply |
[production] |
| 05:39 |
<oblivian@deploy2002> |
helmfile [codfw] START helmfile.d/services/thumbor: apply |
[production] |
| 05:15 |
<hashar@deploy2002> |
Finished deploy [integration/docroot@b816911]: Update Grafana URL (duration: 00m 11s) |
[production] |
| 05:15 |
<hashar@deploy2002> |
Started deploy [integration/docroot@b816911]: Update Grafana URL |
[production] |
| 05:10 |
<joe> |
sudo cumin -b 1 -s 20 'A:swift-fe-codfw' 'systemctl restart swift-proxy.service' |
[production] |
| 04:33 |
<vgutierrez> |
restart haproxy on cp1087 - T334448 |
[production] |
|
2023-04-21
§
|
| 22:12 |
<Operator873> |
granted voice to Fehufanga in #cvn-simplewikis |
[cvn] |
| 22:01 |
<wm-bot> |
<bd808> Update to 7033053 |
[tools.toolviews] |
| 21:58 |
<andrewbogott> |
added raymond-ndibe as project member |
[metricsinfra] |
| 21:36 |
<wm-bot> |
<bd808> Update to 7033053 |
[tools.toolviewstest] |
| 18:27 |
<mvernon@cumin2002> |
END (FAIL) - Cookbook sre.swift.remove-ghost-objects (exit_code=99) from container wikipedia-en-local-public.a8 in codfw |
[production] |
| 18:25 |
<mvernon@cumin2002> |
START - Cookbook sre.swift.remove-ghost-objects from container wikipedia-en-local-public.a8 in codfw |
[production] |
| 15:57 |
<lucaswerkmeister-wmde@deploy2002> |
Finished scap: Backport for [[gerrit:910780|Set wmgUseGraphWithJsonNamespace = true for mediawikiwiki (T124748 T335130)]] (duration: 10m 01s) |
[production] |
| 15:48 |
<lucaswerkmeister-wmde@deploy2002> |
lucaswerkmeister-wmde: Backport for [[gerrit:910780|Set wmgUseGraphWithJsonNamespace = true for mediawikiwiki (T124748 T335130)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 15:47 |
<lucaswerkmeister-wmde@deploy2002> |
Started scap: Backport for [[gerrit:910780|Set wmgUseGraphWithJsonNamespace = true for mediawikiwiki (T124748 T335130)]] |
[production] |
| 14:57 |
<legoktm> |
rebooting codesearch8 to see if that fixes networking issues (T335197) |
[codesearch] |
| 14:44 |
<wm-bot> |
<lucaswerkmeister> Double IRC messages to other bridges |
[tools.bridgebot] |
| 12:18 |
<duesen> |
reverted monky-patch, mwdebug2001 and deploy2002 are back to wmf/1.41.0-wmf.5 (T335183) |
[production] |
| 11:56 |
<duesen> |
monky-patching Ib11a871ff on mwdebug2001 to investigate T335183 |
[production] |
| 10:04 |
<Lucas_WMDE> |
`sudo systemctl reload nginx` on deployment-elastic09, deployment-elastic10, deployment-elastic11 # T335181 |
[releng] |
| 09:03 |
<Amir1> |
finish of the wikibase populate sites table |
[production] |
| 08:35 |
<Amir1> |
start of foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https |
[production] |
| 03:19 |
<eileen> |
civicrm upgraded from 5b63c2b2 to 0fad720a |
[production] |
| 03:11 |
<eileen> |
civicrm upgraded from a2e7c079 to 5b63c2b2 |
[production] |
| 01:41 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host backup2011.codfw.wmnet with OS bullseye |
[production] |
| 01:41 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" |
[production] |
| 01:39 |
<pt1979@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" |
[production] |
| 01:22 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup2011.codfw.wmnet with reason: host reimage |
[production] |
| 01:19 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on backup2011.codfw.wmnet with reason: host reimage |
[production] |
| 00:37 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host backup2010.codfw.wmnet with OS bullseye |
[production] |
| 00:37 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" |
[production] |
| 00:35 |
<pt1979@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" |
[production] |
| 00:18 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup2010.codfw.wmnet with reason: host reimage |
[production] |
| 00:15 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on backup2010.codfw.wmnet with reason: host reimage |
[production] |
| 00:10 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host backup2011.codfw.wmnet with OS bullseye |
[production] |
| 00:00 |
<wm-bot> |
<legoktm> Updated from 398d3de to def9acb |
[tools.poty-stuff] |