2601-2650 of 10000 results (106ms)
2024-07-11 ยง
05:36 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P66256 and previous config saved to /var/cache/conftool/dbconfig/20240711-053659-arnaudb.json [production]
05:29 <marostegui@cumin1002> dbctl commit (dc=all): 'db1163 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P66255 and previous config saved to /var/cache/conftool/dbconfig/20240711-052931-root.json [production]
05:27 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1163 T369514', diff saved to https://phabricator.wikimedia.org/P66254 and previous config saved to /var/cache/conftool/dbconfig/20240711-052702-root.json [production]
05:25 <marostegui@cumin1002> dbctl commit (dc=all): 'Promote db1184 to s1 primary and set section read-write T369514', diff saved to https://phabricator.wikimedia.org/P66253 and previous config saved to /var/cache/conftool/dbconfig/20240711-052540-root.json [production]
05:25 <marostegui@cumin1002> dbctl commit (dc=all): 'Set s1 eqiad as read-only for maintenance - T369514', diff saved to https://phabricator.wikimedia.org/P66252 and previous config saved to /var/cache/conftool/dbconfig/20240711-052507-root.json [production]
05:24 <marostegui> Starting s1 eqiad failover from db1163 to db1184 - T369514 [production]
05:21 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2181 (T367781)', diff saved to https://phabricator.wikimedia.org/P66251 and previous config saved to /var/cache/conftool/dbconfig/20240711-052151-arnaudb.json [production]
05:19 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2181 (T367781)', diff saved to https://phabricator.wikimedia.org/P66250 and previous config saved to /var/cache/conftool/dbconfig/20240711-051941-arnaudb.json [production]
05:19 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2181.codfw.wmnet with reason: Maintenance [production]
05:19 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db2181.codfw.wmnet with reason: Maintenance [production]
05:19 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2167 (T367781)', diff saved to https://phabricator.wikimedia.org/P66249 and previous config saved to /var/cache/conftool/dbconfig/20240711-051920-arnaudb.json [production]
05:04 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P66248 and previous config saved to /var/cache/conftool/dbconfig/20240711-050413-arnaudb.json [production]
04:59 <marostegui@cumin1002> dbctl commit (dc=all): 'Remove db1184 from API/vslow/dump T369514', diff saved to https://phabricator.wikimedia.org/P66247 and previous config saved to /var/cache/conftool/dbconfig/20240711-045905-marostegui.json [production]
04:58 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 36 hosts with reason: Primary switchover s1 T369514 [production]
04:58 <marostegui@cumin1002> dbctl commit (dc=all): 'Set db1184 with weight 0 T369514', diff saved to https://phabricator.wikimedia.org/P66246 and previous config saved to /var/cache/conftool/dbconfig/20240711-045829-marostegui.json [production]
04:58 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 36 hosts with reason: Primary switchover s1 T369514 [production]
04:49 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P66245 and previous config saved to /var/cache/conftool/dbconfig/20240711-044905-arnaudb.json [production]
04:33 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2167 (T367781)', diff saved to https://phabricator.wikimedia.org/P66244 and previous config saved to /var/cache/conftool/dbconfig/20240711-043358-arnaudb.json [production]
04:31 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2167 (T367781)', diff saved to https://phabricator.wikimedia.org/P66243 and previous config saved to /var/cache/conftool/dbconfig/20240711-043147-arnaudb.json [production]
04:31 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2167.codfw.wmnet with reason: Maintenance [production]
04:31 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db2167.codfw.wmnet with reason: Maintenance [production]
04:31 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2166 (T367781)', diff saved to https://phabricator.wikimedia.org/P66242 and previous config saved to /var/cache/conftool/dbconfig/20240711-043124-arnaudb.json [production]
04:16 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P66241 and previous config saved to /var/cache/conftool/dbconfig/20240711-041617-arnaudb.json [production]
04:01 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P66240 and previous config saved to /var/cache/conftool/dbconfig/20240711-040110-arnaudb.json [production]
03:46 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2166 (T367781)', diff saved to https://phabricator.wikimedia.org/P66239 and previous config saved to /var/cache/conftool/dbconfig/20240711-034603-arnaudb.json [production]
03:43 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2166 (T367781)', diff saved to https://phabricator.wikimedia.org/P66238 and previous config saved to /var/cache/conftool/dbconfig/20240711-034352-arnaudb.json [production]
03:43 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2166.codfw.wmnet with reason: Maintenance [production]
03:43 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db2166.codfw.wmnet with reason: Maintenance [production]
03:43 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2165 (T367781)', diff saved to https://phabricator.wikimedia.org/P66237 and previous config saved to /var/cache/conftool/dbconfig/20240711-034330-arnaudb.json [production]
03:28 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P66236 and previous config saved to /var/cache/conftool/dbconfig/20240711-032823-arnaudb.json [production]
03:20 <eileen> civicrm upgraded from 04cb9083 to 3287ced0 [production]
03:13 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P66235 and previous config saved to /var/cache/conftool/dbconfig/20240711-031316-arnaudb.json [production]
03:08 <eileen> civicrm upgraded from 2d1a0aad to 04cb9083 [production]
02:58 <eileen> config revision changed from e02c3a85 to 540f27e6 [production]
02:58 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2165 (T367781)', diff saved to https://phabricator.wikimedia.org/P66234 and previous config saved to /var/cache/conftool/dbconfig/20240711-025809-arnaudb.json [production]
02:55 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2165 (T367781)', diff saved to https://phabricator.wikimedia.org/P66233 and previous config saved to /var/cache/conftool/dbconfig/20240711-025558-arnaudb.json [production]
02:55 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2165.codfw.wmnet with reason: Maintenance [production]
02:55 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db2165.codfw.wmnet with reason: Maintenance [production]
02:55 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2164 (T367781)', diff saved to https://phabricator.wikimedia.org/P66232 and previous config saved to /var/cache/conftool/dbconfig/20240711-025537-arnaudb.json [production]
02:48 <eileen> civicrm upgraded from a17496a2 to 2d1a0aad [production]
02:45 <mutante> stewards2001 - sudo mv /srv/repos/users-db /root/ - run puppet and let it recreate the usersdb repo - this time pulling from gitlab - T369780 T369430 [production]
02:40 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P66231 and previous config saved to /var/cache/conftool/dbconfig/20240711-024030-arnaudb.json [production]
02:25 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P66230 and previous config saved to /var/cache/conftool/dbconfig/20240711-022522-arnaudb.json [production]
02:14 <andrew@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1060.eqiad.wmnet with OS bookworm [production]
02:10 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2164 (T367781)', diff saved to https://phabricator.wikimedia.org/P66229 and previous config saved to /var/cache/conftool/dbconfig/20240711-021015-arnaudb.json [production]
02:08 <eileen> civicrm upgraded from a03085ff to 1e2fcba3 [production]
02:08 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2164 (T367781)', diff saved to https://phabricator.wikimedia.org/P66228 and previous config saved to /var/cache/conftool/dbconfig/20240711-020805-arnaudb.json [production]
02:07 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2186.codfw.wmnet with reason: Maintenance [production]
02:07 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db2186.codfw.wmnet with reason: Maintenance [production]
02:07 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2164.codfw.wmnet with reason: Maintenance [production]