301-350 of 10000 results (97ms)
2024-07-15 §
15:47 <klausman@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
15:45 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1182 (T367781)', diff saved to https://phabricator.wikimedia.org/P66494 and previous config saved to /var/cache/conftool/dbconfig/20240715-154526-arnaudb.json [production]
15:43 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db1182 (T367781)', diff saved to https://phabricator.wikimedia.org/P66493 and previous config saved to /var/cache/conftool/dbconfig/20240715-154312-arnaudb.json [production]
15:43 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1182.eqiad.wmnet with reason: Maintenance [production]
15:42 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db1182.eqiad.wmnet with reason: Maintenance [production]
15:42 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1162 (T367781)', diff saved to https://phabricator.wikimedia.org/P66492 and previous config saved to /var/cache/conftool/dbconfig/20240715-154250-arnaudb.json [production]
15:32 <ayounsi@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on netbox2003.codfw.wmnet with reason: netbox upgrade prep work [production]
15:31 <ayounsi@cumin1002> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on netbox2003.codfw.wmnet with reason: netbox upgrade prep work [production]
15:27 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P66491 and previous config saved to /var/cache/conftool/dbconfig/20240715-152742-arnaudb.json [production]
15:17 <klausman@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
15:16 <klausman@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
15:16 <klausman@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
15:14 <mnz@deploy1002> Finished deploy [airflow-dags/research@5121748]: (no justification provided) (duration: 00m 31s) [production]
15:13 <mnz@deploy1002> Started deploy [airflow-dags/research@5121748]: (no justification provided) [production]
15:13 <klausman@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
15:12 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P66490 and previous config saved to /var/cache/conftool/dbconfig/20240715-151235-arnaudb.json [production]
15:12 <klausman@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
15:12 <klausman@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
15:09 <klausman@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
15:07 <klausman@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
14:57 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1162 (T367781)', diff saved to https://phabricator.wikimedia.org/P66489 and previous config saved to /var/cache/conftool/dbconfig/20240715-145728-arnaudb.json [production]
14:55 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db1162 (T367781)', diff saved to https://phabricator.wikimedia.org/P66488 and previous config saved to /var/cache/conftool/dbconfig/20240715-145517-arnaudb.json [production]
14:55 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1162.eqiad.wmnet with reason: Maintenance [production]
14:55 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db1162.eqiad.wmnet with reason: Maintenance [production]
14:54 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1156 (T367781)', diff saved to https://phabricator.wikimedia.org/P66487 and previous config saved to /var/cache/conftool/dbconfig/20240715-145455-arnaudb.json [production]
14:50 <eevans@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on aqs1013.eqiad.wmnet with reason: Server swap — T362033 [production]
14:49 <eevans@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on aqs1013.eqiad.wmnet with reason: Server swap — T362033 [production]
14:39 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P66486 and previous config saved to /var/cache/conftool/dbconfig/20240715-143948-arnaudb.json [production]
14:24 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P66485 and previous config saved to /var/cache/conftool/dbconfig/20240715-142441-arnaudb.json [production]
14:16 <_joe_> updating conftool to 3.1.0 fleet wide [production]
14:13 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy2005.codfw.wmnet with OS bookworm [production]
14:09 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1156 (T367781)', diff saved to https://phabricator.wikimedia.org/P66484 and previous config saved to /var/cache/conftool/dbconfig/20240715-140934-arnaudb.json [production]
14:07 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db1156 (T367781)', diff saved to https://phabricator.wikimedia.org/P66483 and previous config saved to /var/cache/conftool/dbconfig/20240715-140720-arnaudb.json [production]
14:07 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
14:07 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
14:06 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1156.eqiad.wmnet with reason: Maintenance [production]
14:06 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db1156.eqiad.wmnet with reason: Maintenance [production]
13:58 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy2005.codfw.wmnet with reason: host reimage [production]
13:54 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on dbproxy2005.codfw.wmnet with reason: host reimage [production]
13:53 <oblivian@puppetmaster2001> conftool action : set/pooled=yes; selector: name=mw1386.*,cluster=kubernetes,dc=eqiad [reason: Test conftool sal logging] [production]
13:51 <ayounsi@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on netbox1003.eqiad.wmnet with reason: netbox upgrade prep work [production]
13:51 <ayounsi@cumin1002> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on netbox1003.eqiad.wmnet with reason: netbox upgrade prep work [production]
13:50 <ayounsi@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on netboxdb2003.codfw.wmnet with reason: netbox upgrade prep work [production]
13:50 <ayounsi@cumin1002> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on netboxdb2003.codfw.wmnet with reason: netbox upgrade prep work [production]
13:45 <_joe_> uploading conftool 3.1.0 to bookworm,bullseye,buster [production]
13:41 <Lucas_WMDE> UTC afternoon backport+config window done [production]
13:39 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host dbproxy2005.codfw.wmnet with OS bookworm [production]
13:33 <logmsgbot> lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for [[gerrit:1052699|Add entity-schema to $wgWBRepoSettings['searchIndexTypes'] (T369495)]] (duration: 30m 51s) [production]
13:25 <logmsgbot> lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde: Continuing with sync [production]
13:15 <logmsgbot> lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde: Backport for [[gerrit:1052699|Add entity-schema to $wgWBRepoSettings['searchIndexTypes'] (T369495)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]