1001-1050 of 10000 results (107ms)
2024-04-16 ยง
19:23 <aqu@deploy1002> Started deploy [airflow-dags/analytics_test@9208108]: Regular analytics weekly train [airflow-dags/analytics_test@9208108e] [production]
19:18 <aqu@deploy1002> Started deploy [analytics/refinery@59f7d09]: Regular analytics weekly train [analytics/refinery@59f7d091] [production]
19:17 <aqu> Deployment train for analytics/refinery [production]
19:15 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1246 (T361627)', diff saved to https://phabricator.wikimedia.org/P60687 and previous config saved to /var/cache/conftool/dbconfig/20240416-191522-marostegui.json [production]
19:15 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1246.eqiad.wmnet with reason: Maintenance [production]
19:15 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db1246.eqiad.wmnet with reason: Maintenance [production]
19:14 <hashar@deploy1002> Finished deploy [zuul/deploy@efce3ee]: Redeploy Zuul following host reimaging - T334517 (duration: 00m 13s) [production]
19:14 <hashar@deploy1002> Started deploy [zuul/deploy@efce3ee]: Redeploy Zuul following host reimaging - T334517 [production]
19:12 <hashar@deploy1002> Finished deploy [zuul/deploy@efce3ee]: Redeploy Zuul following host reimaging - T334517 (duration: 00m 03s) [production]
19:12 <hashar@deploy1002> Started deploy [zuul/deploy@efce3ee]: Redeploy Zuul following host reimaging - T334517 [production]
19:11 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1239.eqiad.wmnet with reason: Maintenance [production]
19:11 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db1239.eqiad.wmnet with reason: Maintenance [production]
19:11 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1233 (T361627)', diff saved to https://phabricator.wikimedia.org/P60686 and previous config saved to /var/cache/conftool/dbconfig/20240416-191128-marostegui.json [production]
19:08 <dancy@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.43.0-wmf.1 refs T361395 [production]
18:56 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P60685 and previous config saved to /var/cache/conftool/dbconfig/20240416-185621-marostegui.json [production]
18:52 <aqu@deploy1002> Finished deploy [airflow-dags/analytics@9208108]: Regular analytics weekly train [airflow-dags/analytics@9208108e] (duration: 00m 26s) [production]
18:52 <aqu@deploy1002> Started deploy [airflow-dags/analytics@9208108]: Regular analytics weekly train [airflow-dags/analytics@9208108e] [production]
18:50 <dancy@deploy1002> Installation of scap version "4.77.0" completed for 340 hosts [production]
18:49 <dancy@deploy1002> Installing scap version "4.77.0" for 340 hosts [production]
18:48 <dancy@deploy1002> Finished scap: Backport for [[gerrit:1020231|[Parser] Temporarily disable deprecation warnings for dynamic properties (T362692)]] (duration: 22m 56s) [production]
18:44 <dzahn@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host contint1002.wikimedia.org with OS bullseye [production]
18:41 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P60684 and previous config saved to /var/cache/conftool/dbconfig/20240416-184113-marostegui.json [production]
18:40 <mutante> contint1002 - sudo a2dismod mpm_event to work around known race condition and fix failed initial puppet run - T334517 [production]
18:35 <dancy@deploy1002> cscott and dancy: Continuing with sync [production]
18:29 <bearloga@deploy1002> Finished deploy [airflow-dags/analytics_product@77af7cb]: (no justification provided) (duration: 00m 07s) [production]
18:29 <bearloga@deploy1002> Started deploy [airflow-dags/analytics_product@77af7cb]: (no justification provided) [production]
18:29 <dancy@deploy1002> cscott and dancy: Backport for [[gerrit:1020231|[Parser] Temporarily disable deprecation warnings for dynamic properties (T362692)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
18:26 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1233 (T361627)', diff saved to https://phabricator.wikimedia.org/P60683 and previous config saved to /var/cache/conftool/dbconfig/20240416-182606-marostegui.json [production]
18:26 <dancy@deploy1002> Started scap: Backport for [[gerrit:1020231|[Parser] Temporarily disable deprecation warnings for dynamic properties (T362692)]] [production]
18:10 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1233 (T361627)', diff saved to https://phabricator.wikimedia.org/P60682 and previous config saved to /var/cache/conftool/dbconfig/20240416-181001-marostegui.json [production]
18:09 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1233.eqiad.wmnet with reason: Maintenance [production]
18:09 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db1233.eqiad.wmnet with reason: Maintenance [production]
18:09 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1229 (T361627)', diff saved to https://phabricator.wikimedia.org/P60681 and previous config saved to /var/cache/conftool/dbconfig/20240416-180938-marostegui.json [production]
18:07 <dzahn@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on contint1002.wikimedia.org with reason: host reimage [production]
18:04 <dzahn@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on contint1002.wikimedia.org with reason: host reimage [production]
17:54 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P60680 and previous config saved to /var/cache/conftool/dbconfig/20240416-175431-marostegui.json [production]
17:52 <dzahn@cumin2002> START - Cookbook sre.hosts.reimage for host contint1002.wikimedia.org with OS bullseye [production]
17:51 <mutante> CI - jenkins on contint1002 disabled - reimaging in progress [production]
17:50 <bearloga@deploy1002> Finished deploy [airflow-dags/analytics_product@bb33843]: (no justification provided) (duration: 00m 06s) [production]
17:50 <bearloga@deploy1002> Started deploy [airflow-dags/analytics_product@bb33843]: (no justification provided) [production]
17:48 <dzahn@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on contint1002.wikimedia.org with reason: reimage https://phabricator.wikmedia.org/T334517 [production]
17:48 <dzahn@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on contint1002.wikimedia.org with reason: reimage https://phabricator.wikmedia.org/T334517 [production]
17:39 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P60679 and previous config saved to /var/cache/conftool/dbconfig/20240416-173923-marostegui.json [production]
17:37 <mutante> CI - disabling zuul-merger on contint1002 - there is another on contint2002 [production]
17:35 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcontrol2009-dev.mgmt.codfw.wmnet with reboot policy FORCED [production]
17:25 <marostegui@cumin1002> dbctl commit (dc=all): 'db2127 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60678 and previous config saved to /var/cache/conftool/dbconfig/20240416-172515-root.json [production]
17:24 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1229 (T361627)', diff saved to https://phabricator.wikimedia.org/P60677 and previous config saved to /var/cache/conftool/dbconfig/20240416-172415-marostegui.json [production]
17:22 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1229 (T361627)', diff saved to https://phabricator.wikimedia.org/P60676 and previous config saved to /var/cache/conftool/dbconfig/20240416-172201-marostegui.json [production]
17:21 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1229.eqiad.wmnet with reason: Maintenance [production]
17:21 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db1229.eqiad.wmnet with reason: Maintenance [production]