5301-5350 of 10000 results (93ms)
2024-02-09 §
10:23 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance [production]
10:23 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance [production]
10:23 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1185 (T352010)', diff saved to https://phabricator.wikimedia.org/P56581 and previous config saved to /var/cache/conftool/dbconfig/20240209-102314-ladsgroup.json [production]
10:08 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P56580 and previous config saved to /var/cache/conftool/dbconfig/20240209-100808-ladsgroup.json [production]
09:54 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2194.codfw.wmnet with OS bookworm [production]
09:53 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P56579 and previous config saved to /var/cache/conftool/dbconfig/20240209-095301-ladsgroup.json [production]
09:46 <moritzm> uploaded openjdk-8 8u402-ga-2~deb10u1 for buster-wikimedia (backport of latest Java 8 security updates) [production]
09:37 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1185 (T352010)', diff saved to https://phabricator.wikimedia.org/P56578 and previous config saved to /var/cache/conftool/dbconfig/20240209-093754-ladsgroup.json [production]
09:32 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2194.codfw.wmnet with reason: host reimage [production]
09:28 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2194.codfw.wmnet with reason: host reimage [production]
09:08 <arnaudb@cumin1002> START - Cookbook sre.hosts.reimage for host db2194.codfw.wmnet with OS bookworm [production]
08:39 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts puppetmaster2003.codfw.wmnet [production]
08:39 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:39 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: puppetmaster2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
08:37 <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: puppetmaster2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
08:35 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
08:29 <jmm@cumin2002> START - Cookbook sre.hosts.decommission for hosts puppetmaster2003.codfw.wmnet [production]
06:52 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1185 (T352010)', diff saved to https://phabricator.wikimedia.org/P56577 and previous config saved to /var/cache/conftool/dbconfig/20240209-065147-ladsgroup.json [production]
06:51 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance [production]
06:51 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance [production]
06:51 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1161 (T352010)', diff saved to https://phabricator.wikimedia.org/P56576 and previous config saved to /var/cache/conftool/dbconfig/20240209-065125-ladsgroup.json [production]
06:38 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1124.eqiad.wmnet [production]
06:38 <marostegui@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
06:38 <marostegui@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1124.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002" [production]
06:36 <marostegui@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1124.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002" [production]
06:36 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P56575 and previous config saved to /var/cache/conftool/dbconfig/20240209-063618-ladsgroup.json [production]
06:34 <marostegui@cumin1002> START - Cookbook sre.dns.netbox [production]
06:29 <marostegui@cumin1002> START - Cookbook sre.hosts.decommission for hosts db1124.eqiad.wmnet [production]
06:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P56574 and previous config saved to /var/cache/conftool/dbconfig/20240209-062111-ladsgroup.json [production]
06:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1161 (T352010)', diff saved to https://phabricator.wikimedia.org/P56573 and previous config saved to /var/cache/conftool/dbconfig/20240209-060605-ladsgroup.json [production]
05:48 <marostegui> dbmaint Schema change on s7@codfw T357067 [production]
04:50 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
04:49 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
03:00 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1161 (T352010)', diff saved to https://phabricator.wikimedia.org/P56572 and previous config saved to /var/cache/conftool/dbconfig/20240209-030028-ladsgroup.json [production]
03:00 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
02:59 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
02:59 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance [production]
02:59 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance [production]
00:05 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1145.eqiad.wmnet with reason: Maintenance [production]
00:04 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1145.eqiad.wmnet with reason: Maintenance [production]
2024-02-08 §
23:57 <volans@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest1001.mgmt.eqiad.wmnet with reboot policy GRACEFUL [production]
23:56 <volans@cumin1002> START - Cookbook sre.hosts.provision for host sretest1001.mgmt.eqiad.wmnet with reboot policy GRACEFUL [production]
23:50 <foks> removing 14 files for legal compliance [production]
23:28 <foks> removing one file for legal compliance [production]
23:17 <foks> removing two files for legal compliance [production]
22:58 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Unbanning all hosts in cloudelastic [production]
22:57 <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Unbanning all hosts in cloudelastic [production]
22:51 <jhathaway> made a stupid mistake and accidentally installed knot & unbound on dns1004, based on logs I don't think any harm was caused, they have since been removed [production]
22:44 <jclark@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:44 <jclark@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: racked and provision network restbase servers - jclark@cumin1002" [production]