|
2024-09-18
§
|
| 05:33 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'T375047', diff saved to https://phabricator.wikimedia.org/P69242 and previous config saved to /var/cache/conftool/dbconfig/20240918-053357-arnaudb.json |
[production] |
| 05:31 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Promote db2214 to s6 primary T375047', diff saved to https://phabricator.wikimedia.org/P69241 and previous config saved to /var/cache/conftool/dbconfig/20240918-053115-arnaudb.json |
[production] |
| 05:30 |
<arnaudb> |
Starting s6 codfw failover from db2129 to db2214 - T375047 |
[production] |
| 05:24 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Set db2214 with weight 0 T375047', diff saved to https://phabricator.wikimedia.org/P69240 and previous config saved to /var/cache/conftool/dbconfig/20240918-052446-arnaudb.json |
[production] |
| 05:24 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s6 T375047 |
[production] |
| 05:23 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 25 hosts with reason: Primary switchover s6 T375047 |
[production] |
| 03:56 |
<tchin> |
Deployed refinery-source using jenkins |
[analytics] |
| 00:11 |
<dzahn@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on gitlab2002.wikimedia.org with reason: version upgrade |
[production] |
| 00:11 |
<dzahn@cumin2002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on gitlab2002.wikimedia.org with reason: version upgrade |
[production] |
|
2024-09-17
§
|
| 23:56 |
<dzahn@cumin2002> |
END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab2002.wikimedia.org with reason: security release 20240917 |
[production] |
| 21:25 |
<sbassett> |
Deployed mitigation for T374438 |
[production] |
| 21:24 |
<wmbot~dcaro@urcuchillay> |
END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) (T374043) |
[admin] |
| 21:21 |
<hashar> |
UTC late backport window is completed |
[production] |
| 21:18 |
<dzahn@cumin2002> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: security release 20240917 |
[production] |
| 21:07 |
<dzahn@cumin2002> |
END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: security release 20240917 |
[production] |
| 20:56 |
<hashar@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1073408|logging: rm per channel 'error' logging (T228838)]] (duration: 08m 22s) |
[production] |
| 20:52 |
<hashar@deploy1003> |
hashar: Continuing with sync |
[production] |
| 20:50 |
<hashar@deploy1003> |
hashar: Backport for [[gerrit:1073408|logging: rm per channel 'error' logging (T228838)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 20:48 |
<dzahn@cumin2002> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: security release 20240917 |
[production] |
| 20:48 |
<hashar@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1073408|logging: rm per channel 'error' logging (T228838)]] |
[production] |
| 20:46 |
<hashar@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1071712|Improve $wgFooterIcons override, simplify $wmgWikimediaIcon]] (duration: 08m 05s) |
[production] |
| 20:45 |
<bd808@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/toolhub: apply |
[production] |
| 20:44 |
<bd808@deploy1003> |
helmfile [eqiad] START helmfile.d/services/toolhub: apply |
[production] |
| 20:43 |
<bd808@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/toolhub: apply |
[production] |
| 20:42 |
<dzahn@cumin2002> |
END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: security release 20240917 |
[production] |
| 20:42 |
<bd808@deploy1003> |
helmfile [codfw] START helmfile.d/services/toolhub: apply |
[production] |
| 20:42 |
<hashar@deploy1003> |
matmarex, hashar: Continuing with sync |
[production] |
| 20:42 |
<bd808@deploy1003> |
helmfile [staging] DONE helmfile.d/services/toolhub: apply |
[production] |
| 20:41 |
<bd808@deploy1003> |
helmfile [staging] START helmfile.d/services/toolhub: apply |
[production] |
| 20:40 |
<hashar@deploy1003> |
matmarex, hashar: Backport for [[gerrit:1071712|Improve $wgFooterIcons override, simplify $wmgWikimediaIcon]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 20:38 |
<hashar@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1071712|Improve $wgFooterIcons override, simplify $wmgWikimediaIcon]] |
[production] |
| 20:37 |
<hashar@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1067390|Revert "Enter deprecation trial for third-party cookie blocking" (T359957)]] (duration: 06m 37s) |
[production] |
| 20:32 |
<hashar@deploy1003> |
tgr, hashar: Continuing with sync |
[production] |
| 20:32 |
<hashar@deploy1003> |
tgr, hashar: Backport for [[gerrit:1067390|Revert "Enter deprecation trial for third-party cookie blocking" (T359957)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 20:32 |
<dzahn@cumin2002> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: security release 20240917 |
[production] |
| 20:30 |
<hashar@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1067390|Revert "Enter deprecation trial for third-party cookie blocking" (T359957)]] |
[production] |
| 20:27 |
<hashar@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:649942|Assign the API portal to the Wikimedia group for CentralNotice (T270308)]] (duration: 12m 11s) |
[production] |
| 20:22 |
<hashar@deploy1003> |
ejegg, hashar: Continuing with sync |
[production] |
| 20:17 |
<hashar@deploy1003> |
ejegg, hashar: Backport for [[gerrit:649942|Assign the API portal to the Wikimedia group for CentralNotice (T270308)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 20:14 |
<hashar@deploy1003> |
Started scap sync-world: Backport for [[gerrit:649942|Assign the API portal to the Wikimedia group for CentralNotice (T270308)]] |
[production] |
| 20:07 |
<dcausse@deploy1003> |
helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 20:07 |
<dcausse@deploy1003> |
helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 20:07 |
<dcausse@deploy1003> |
helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 20:00 |
<wmbot~lucaswerkmeister@tools-bastion-13> |
deployed 4cfe24a186 (fix background runner database config) |
[tools.quickcategories] |
| 19:58 |
<wmbot~lucaswerkmeister@tools-bastion-13> |
kubectl delete deployment background-runner && kubectl create -f deployment.yaml # recreate background runner, seemed to be running old code version? idk |
[tools.quickcategories] |
| 19:54 |
<wmbot~lucaswerkmeister@tools-bastion-13> |
deployed 0be765159c (editgroups from envvars) |
[tools.quickcategories] |
| 19:53 |
<wmbot~lucaswerkmeister@tools-bastion-13> |
toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__SINCE 2021-09-14T00:00:00Z |
[tools.quickcategories] |
| 19:52 |
<wmbot~lucaswerkmeister@tools-bastion-13> |
toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__URL "'https://editgroups-commons.toolforge.org/b/QC/{0}/'" |
[tools.quickcategories] |
| 19:52 |
<wmbot~lucaswerkmeister@tools-bastion-13> |
toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__DOMAIN commons.wikimedia.org |
[tools.quickcategories] |
| 19:49 |
<herron> |
pyrra upgraded to 0.7.7-1 |
[production] |