351-400 of 10000 results (77ms)
2024-05-23 §
07:20 <dcausse@deploy1002> Finished scap: Backport for [[gerrit:1035061|cirrus: Keep archive writes running through cirrus]] (duration: 17m 19s) [production]
07:08 <dcausse@deploy1002> ebernhardson and dcausse: Continuing with sync [production]
07:06 <dcausse@deploy1002> ebernhardson and dcausse: Backport for [[gerrit:1035061|cirrus: Keep archive writes running through cirrus]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
07:03 <dcausse@deploy1002> Started scap: Backport for [[gerrit:1035061|cirrus: Keep archive writes running through cirrus]] [production]
06:45 <dcausse@deploy1002> Finished deploy [airflow-dags/search@49369da]: search: automate graph split and n3 dump generation (duration: 00m 19s) [production]
06:45 <dcausse@deploy1002> Started deploy [airflow-dags/search@49369da]: search: automate graph split and n3 dump generation [production]
06:40 <marostegui@cumin1002> dbctl commit (dc=all): 'db1174 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P62954 and previous config saved to /var/cache/conftool/dbconfig/20240523-064027-root.json [production]
06:31 <jelto@cumin1002> END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Replica to new version [production]
06:30 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depool db1238 T363689', diff saved to https://phabricator.wikimedia.org/P62953 and previous config saved to /var/cache/conftool/dbconfig/20240523-063025-arnaudb.json [production]
06:25 <marostegui@cumin1002> dbctl commit (dc=all): 'db1174 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P62952 and previous config saved to /var/cache/conftool/dbconfig/20240523-062521-root.json [production]
06:24 <jelto@cumin1002> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Replica to new version [production]
06:24 <jelto@cumin1002> END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab Replica to new version [production]
06:16 <jelto@cumin1002> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab Replica to new version [production]
06:15 <arnaudb@cumin1002> dbctl commit (dc=all): 'Promote db1160 to s4 primary and set section read-write T363689', diff saved to https://phabricator.wikimedia.org/P62951 and previous config saved to /var/cache/conftool/dbconfig/20240523-061524-arnaudb.json [production]
06:14 <arnaudb@cumin1002> dbctl commit (dc=all): 'Set s4 eqiad as read-only for maintenance - T363689', diff saved to https://phabricator.wikimedia.org/P62950 and previous config saved to /var/cache/conftool/dbconfig/20240523-061408-arnaudb.json [production]
06:13 <arnaudb> Starting s4 eqiad failover from db1238 to db1160 - T363689 [production]
06:10 <marostegui@cumin1002> dbctl commit (dc=all): 'db1174 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P62949 and previous config saved to /var/cache/conftool/dbconfig/20240523-061014-root.json [production]
05:57 <marostegui@cumin1002> dbctl commit (dc=all): 'db1226 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P62948 and previous config saved to /var/cache/conftool/dbconfig/20240523-055747-root.json [production]
05:56 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1155.eqiad.wmnet with OS bookworm [production]
05:55 <marostegui@cumin1002> dbctl commit (dc=all): 'db1174 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P62947 and previous config saved to /var/cache/conftool/dbconfig/20240523-055508-root.json [production]
05:48 <arnaudb@cumin1002> dbctl commit (dc=all): 'Set db1160 with weight 0 T363689', diff saved to https://phabricator.wikimedia.org/P62946 and previous config saved to /var/cache/conftool/dbconfig/20240523-054816-arnaudb.json [production]
05:47 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 33 hosts with reason: Primary switchover s4 T363689 [production]
05:47 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 33 hosts with reason: Primary switchover s4 T363689 [production]
05:42 <marostegui@cumin1002> dbctl commit (dc=all): 'db1226 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P62945 and previous config saved to /var/cache/conftool/dbconfig/20240523-054240-root.json [production]
05:40 <marostegui@cumin1002> dbctl commit (dc=all): 'db1174 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P62944 and previous config saved to /var/cache/conftool/dbconfig/20240523-054002-root.json [production]
05:35 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1155.eqiad.wmnet with reason: host reimage [production]
05:32 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db1155.eqiad.wmnet with reason: host reimage [production]
05:27 <marostegui@cumin1002> dbctl commit (dc=all): 'db1226 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P62942 and previous config saved to /var/cache/conftool/dbconfig/20240523-052734-root.json [production]
05:24 <marostegui@cumin1002> dbctl commit (dc=all): 'db1174 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P62941 and previous config saved to /var/cache/conftool/dbconfig/20240523-052456-root.json [production]
05:17 <marostegui@cumin1002> START - Cookbook sre.hosts.reimage for host db1155.eqiad.wmnet with OS bookworm [production]
05:12 <marostegui@cumin1002> dbctl commit (dc=all): 'db1226 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P62940 and previous config saved to /var/cache/conftool/dbconfig/20240523-051228-root.json [production]
05:09 <marostegui@cumin1002> dbctl commit (dc=all): 'db1174 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P62939 and previous config saved to /var/cache/conftool/dbconfig/20240523-050950-root.json [production]
05:08 <marostegui> Install 10..6.18 on db1174 T365338 [production]
05:06 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1174', diff saved to https://phabricator.wikimedia.org/P62938 and previous config saved to /var/cache/conftool/dbconfig/20240523-050626-root.json [production]
04:57 <marostegui@cumin1002> dbctl commit (dc=all): 'db1226 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P62937 and previous config saved to /var/cache/conftool/dbconfig/20240523-045722-root.json [production]
03:16 <eileen> civicrm upgraded from 50211434 to 252eed3c [production]
02:56 <eileen> config revision changed from f8af8188 to d8905b73 [production]
02:54 <eileen> config revision changed from b9fbe283 to f8af8188 [production]
02:53 <eileen> tools upgraded from ad48f63e to bce5f52b [production]
01:50 <eileen> civicrm upgraded from 5cb7c467 to 50211434 [production]
01:26 <eileen> civicrm upgraded from 172feea2 to 5cb7c467 [production]
00:29 <ejegg> fundraising civicrm upgraded from 84c36324 to 172feea2 [production]
2024-05-22 §
23:49 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1214 (T364299)', diff saved to https://phabricator.wikimedia.org/P62936 and previous config saved to /var/cache/conftool/dbconfig/20240522-234937-marostegui.json [production]
23:49 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance [production]
23:49 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance [production]
23:49 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1211 (T364299)', diff saved to https://phabricator.wikimedia.org/P62935 and previous config saved to /var/cache/conftool/dbconfig/20240522-234914-marostegui.json [production]
23:34 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P62934 and previous config saved to /var/cache/conftool/dbconfig/20240522-233406-marostegui.json [production]
23:18 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P62933 and previous config saved to /var/cache/conftool/dbconfig/20240522-231858-marostegui.json [production]
23:03 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1211 (T364299)', diff saved to https://phabricator.wikimedia.org/P62932 and previous config saved to /var/cache/conftool/dbconfig/20240522-230350-marostegui.json [production]
22:24 <ryankemper@cumin2002> END (PASS) - Cookbook sre.hadoop.roll-restart-masters (exit_code=0) restart masters for Hadoop test cluster: Restart of jvm daemons. [production]