1701-1750 of 10000 results (81ms)
2023-02-10 §
02:07 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
02:07 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new mw nodes row D - pt1979@cumin2002" [production]
02:06 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host mw2443.mgmt.codfw.wmnet with reboot policy FORCED [production]
02:06 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new mw nodes row D - pt1979@cumin2002" [production]
02:06 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2441.mgmt.codfw.wmnet with reboot policy FORCED [production]
02:05 <zabe> deployed mitigations for T326691 [production]
02:03 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
02:01 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host mw2442.mgmt.codfw.wmnet with reboot policy FORCED [production]
02:00 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mw2441.mgmt.codfw.wmnet with reboot policy FORCED [production]
02:00 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host mw2441.mgmt.codfw.wmnet with reboot policy FORCED [production]
02:00 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host mw2441.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:58 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2440.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:51 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2439.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:50 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host mw2440.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:49 <Reedy> creating wbc_entity_usage on foundationwiki - T321967 [production]
01:49 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mw2438.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:43 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host mw2439.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:43 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host mw2438.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:42 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2437.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:42 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2436.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:35 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host mw2437.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:34 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host mw2436.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:32 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
01:32 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new mw nodes in row C - pt1979@cumin2002" [production]
01:31 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for new mw nodes in row C - pt1979@cumin2002" [production]
01:29 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
2023-02-09 §
23:49 <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:887858|Revert "Start reading from rev_comment_id in cebwiki"]] (duration: 09m 04s) [production]
23:41 <ladsgroup@deploy1002> ladsgroup: Backport for [[gerrit:887858|Revert "Start reading from rev_comment_id in cebwiki"]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
23:39 <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:887858|Revert "Start reading from rev_comment_id in cebwiki"]] [production]
23:15 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2177 (T328255)', diff saved to https://phabricator.wikimedia.org/P44114 and previous config saved to /var/cache/conftool/dbconfig/20230209-231522-ladsgroup.json [production]
23:00 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P44113 and previous config saved to /var/cache/conftool/dbconfig/20230209-230016-ladsgroup.json [production]
22:50 <zabe@deploy1002> Finished scap: Backport for [[gerrit:888093|Start reading from rev_comment_id in cebwiki (T275246)]] (duration: 08m 26s) [production]
22:45 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P44112 and previous config saved to /var/cache/conftool/dbconfig/20230209-224509-ladsgroup.json [production]
22:44 <zabe@deploy1002> zabe: Backport for [[gerrit:888093|Start reading from rev_comment_id in cebwiki (T275246)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
22:42 <zabe@deploy1002> Started scap: Backport for [[gerrit:888093|Start reading from rev_comment_id in cebwiki (T275246)]] [production]
22:30 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2177 (T328255)', diff saved to https://phabricator.wikimedia.org/P44111 and previous config saved to /var/cache/conftool/dbconfig/20230209-223003-ladsgroup.json [production]
22:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2177 (T328255)', diff saved to https://phabricator.wikimedia.org/P44110 and previous config saved to /var/cache/conftool/dbconfig/20230209-222137-ladsgroup.json [production]
22:21 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2177.codfw.wmnet with reason: Maintenance [production]
22:21 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2177.codfw.wmnet with reason: Maintenance [production]
22:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2156 (T328255)', diff saved to https://phabricator.wikimedia.org/P44109 and previous config saved to /var/cache/conftool/dbconfig/20230209-222126-ladsgroup.json [production]
22:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P44108 and previous config saved to /var/cache/conftool/dbconfig/20230209-220620-ladsgroup.json [production]
21:51 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P44107 and previous config saved to /var/cache/conftool/dbconfig/20230209-215114-ladsgroup.json [production]
21:36 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2156 (T328255)', diff saved to https://phabricator.wikimedia.org/P44106 and previous config saved to /var/cache/conftool/dbconfig/20230209-213607-ladsgroup.json [production]
21:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2156 (T328255)', diff saved to https://phabricator.wikimedia.org/P44105 and previous config saved to /var/cache/conftool/dbconfig/20230209-212747-ladsgroup.json [production]
21:27 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2094.codfw.wmnet with reason: Maintenance [production]
21:27 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2094.codfw.wmnet with reason: Maintenance [production]
21:27 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance [production]
21:27 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance [production]
21:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2149 (T328255)', diff saved to https://phabricator.wikimedia.org/P44104 and previous config saved to /var/cache/conftool/dbconfig/20230209-212732-ladsgroup.json [production]
21:19 <thcipriani@deploy1002> Sync cancelled. [production]