301-350 of 10000 results (74ms)
2023-02-08 ยง
10:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P43793 and previous config saved to /var/cache/conftool/dbconfig/20230208-105001-marostegui.json [production]
10:49 <zabe@deploy1002> Started scap: Backport for [[gerrit:887748|Remove cul_reason comment table migration code (T233004 T329151)]] [production]
10:38 <jmm@cumin2002> END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe [production]
10:35 <jmm@cumin2002> START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe [production]
10:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P43791 and previous config saved to /var/cache/conftool/dbconfig/20230208-103455-marostegui.json [production]
10:33 <volans> deploying python3-wmflib_1.2.1 to the fleet [production]
10:28 <zabe@deploy1002> Finished scap: Backport for [[gerrit:887747|Revert "slwiki: Raise AF emergency disable treshold+count" (T328366)]] (duration: 08m 49s) [production]
10:26 <marostegui> Failover m2-master from dbproxy1013 to dbproxy1015 T329073 [production]
10:21 <zabe@deploy1002> zabe: Backport for [[gerrit:887747|Revert "slwiki: Raise AF emergency disable treshold+count" (T328366)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
10:19 <zabe@deploy1002> Started scap: Backport for [[gerrit:887747|Revert "slwiki: Raise AF emergency disable treshold+count" (T328366)]] [production]
10:19 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T328817)', diff saved to https://phabricator.wikimedia.org/P43790 and previous config saved to /var/cache/conftool/dbconfig/20230208-101948-marostegui.json [production]
10:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1113:3316 (T328817)', diff saved to https://phabricator.wikimedia.org/P43789 and previous config saved to /var/cache/conftool/dbconfig/20230208-101534-marostegui.json [production]
10:15 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1113.eqiad.wmnet with reason: Maintenance [production]
10:15 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1113.eqiad.wmnet with reason: Maintenance [production]
10:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T328817)', diff saved to https://phabricator.wikimedia.org/P43788 and previous config saved to /var/cache/conftool/dbconfig/20230208-101512-marostegui.json [production]
10:08 <phedenskog@deploy1002> Finished deploy [performance/navtiming@079891a]: (no justification provided) (duration: 00m 08s) [production]
10:08 <phedenskog@deploy1002> Started deploy [performance/navtiming@079891a]: (no justification provided) [production]
10:07 <jelto@cumin1001> END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1003.wikimedia.org with reason: Test Upgrade GitLab Replica gitlab1003 with invalid version [production]
10:07 <jelto@cumin1001> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Test Upgrade GitLab Replica gitlab1003 with invalid version [production]
10:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P43787 and previous config saved to /var/cache/conftool/dbconfig/20230208-100006-marostegui.json [production]
09:59 <moritzm> installing openssl security updates on bullseye [production]
09:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db1096 (s5,s6) from dbctl T329147', diff saved to https://phabricator.wikimedia.org/P43786 and previous config saved to /var/cache/conftool/dbconfig/20230208-095207-marostegui.json [production]
09:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P43785 and previous config saved to /var/cache/conftool/dbconfig/20230208-094500-marostegui.json [production]
09:29 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T328817)', diff saved to https://phabricator.wikimedia.org/P43783 and previous config saved to /var/cache/conftool/dbconfig/20230208-092954-marostegui.json [production]
09:14 <godog> purge user_auth table on grafana1002 - T328784 [production]
08:54 <moritzm> installing imagemagick security updates [production]
08:29 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1098:3316 (T328817)', diff saved to https://phabricator.wikimedia.org/P43782 and previous config saved to /var/cache/conftool/dbconfig/20230208-082938-marostegui.json [production]
08:29 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1098.eqiad.wmnet with reason: Maintenance [production]
08:29 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1098.eqiad.wmnet with reason: Maintenance [production]
08:29 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T328817)', diff saved to https://phabricator.wikimedia.org/P43781 and previous config saved to /var/cache/conftool/dbconfig/20230208-082916-marostegui.json [production]
08:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P43780 and previous config saved to /var/cache/conftool/dbconfig/20230208-081410-marostegui.json [production]
08:01 <marostegui> dbmaint deploy schema change on s3 eqiad (with replication) T328807 T328828 [production]
07:59 <marostegui> dbmaint deploy schema change on s1 eqiad (with replication) T328807 T328828 [production]
07:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P43779 and previous config saved to /var/cache/conftool/dbconfig/20230208-075903-marostegui.json [production]
07:56 <marostegui> dbmaint deploy schema change on s7 eqiad (with replication) T328807 T328828 [production]
07:43 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T328817)', diff saved to https://phabricator.wikimedia.org/P43778 and previous config saved to /var/cache/conftool/dbconfig/20230208-074357-marostegui.json [production]
07:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1096:3316 (T328817)', diff saved to https://phabricator.wikimedia.org/P43777 and previous config saved to /var/cache/conftool/dbconfig/20230208-073837-marostegui.json [production]
07:38 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1096.eqiad.wmnet with reason: Maintenance [production]
07:38 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1096.eqiad.wmnet with reason: Maintenance [production]
07:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1173 (T328817)', diff saved to https://phabricator.wikimedia.org/P43776 and previous config saved to /var/cache/conftool/dbconfig/20230208-073816-marostegui.json [production]
07:38 <marostegui> dbmaint deploy schema change on s2 eqiad (with replication) T328807 T328828 [production]
07:23 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P43775 and previous config saved to /var/cache/conftool/dbconfig/20230208-072310-marostegui.json [production]
07:19 <marostegui> dbmaint deploy schema change on s5 eqiad (with replication) T328807 T328828 [production]
07:18 <marostegui> dbmaint deploy schema change on s4 eqiad (with replication) T328807 T328828 [production]
07:18 <marostegui> dbmaint deploy schema change on s8 eqiad (with replication) T328807 T328828 [production]
07:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P43774 and previous config saved to /var/cache/conftool/dbconfig/20230208-070803-marostegui.json [production]
07:07 <marostegui> Install 10.6.12 on pc2014 T329011 [production]
06:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1173 (T328817)', diff saved to https://phabricator.wikimedia.org/P43773 and previous config saved to /var/cache/conftool/dbconfig/20230208-065257-marostegui.json [production]
06:52 <marostegui@deploy1002> Finished scap: Backport for [[gerrit:887445|ProductionServices.php: Promote pc2011 back to pc1 master]] (duration: 16m 01s) [production]
06:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1173 (T328817)', diff saved to https://phabricator.wikimedia.org/P43772 and previous config saved to /var/cache/conftool/dbconfig/20230208-065149-marostegui.json [production]