2651-2700 of 10000 results (93ms)
2023-10-02 §
11:49 <hnowlan@deploy2002> helmfile [codfw] START helmfile.d/admin 'apply'. [production]
11:47 <aborrero@cumin1001> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) bastion.bastioninfra-codfw1dev.codfw1dev.wmcloud.org on all recursors [production]
11:47 <aborrero@cumin1001> START - Cookbook sre.dns.wipe-cache bastion.bastioninfra-codfw1dev.codfw1dev.wmcloud.org on all recursors [production]
11:46 <hnowlan@deploy2002> helmfile [eqiad] DONE helmfile.d/admin 'apply'. [production]
11:45 <hnowlan@deploy2002> helmfile [eqiad] START helmfile.d/admin 'apply'. [production]
11:42 <hnowlan@deploy2002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
11:42 <hnowlan@deploy2002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
11:40 <hnowlan@deploy2002> helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [production]
11:38 <hnowlan@deploy2002> helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [production]
11:35 <jiji@deploy2002> helmfile [eqiad] START helmfile.d/services/push-notifications: apply [production]
10:58 <jiji@deploy2002> helmfile [codfw] DONE helmfile.d/services/push-notifications: apply [production]
10:58 <jiji@deploy2002> helmfile [codfw] START helmfile.d/services/push-notifications: apply [production]
10:55 <jiji@deploy2002> helmfile [staging] DONE helmfile.d/services/push-notifications: apply [production]
10:54 <jiji@deploy2002> helmfile [staging] START helmfile.d/services/push-notifications: apply [production]
10:49 <fabfur> swap purged on cp4040 to use UDS instead of TCP for Varnish (T347837) [production]
10:43 <jiji@deploy2002> helmfile [staging] DONE helmfile.d/services/push-notifications: apply [production]
10:43 <jiji@deploy2002> helmfile [staging] START helmfile.d/services/push-notifications: apply [production]
10:34 <fabfur> depool cp4040 to test new purged version (T347837) [production]
09:48 <cmooney@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "add codfw new switches - cmooney@cumin1001" [production]
09:47 <cmooney@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "add codfw new switches - cmooney@cumin1001" [production]
09:06 <taavi@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcontrol1006.eqiad.wmnet with OS bullseye [production]
08:34 <taavi@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcontrol1006.eqiad.wmnet with reason: host reimage [production]
08:31 <taavi@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcontrol1006.eqiad.wmnet with reason: host reimage [production]
08:24 <aikochou@deploy2002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . [production]
08:21 <taavi@cumin1001> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcontrol1006 [production]
08:21 <taavi@cumin1001> START - Cookbook sre.network.configure-switch-interfaces for host cloudcontrol1006 [production]
08:18 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2098.codfw.wmnet with reason: Maintenance [production]
08:17 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2098.codfw.wmnet with reason: Maintenance [production]
07:49 <godog> +150G to prometheus@k8s in codfw [production]
07:47 <taavi@cumin1001> START - Cookbook sre.hosts.reimage for host cloudcontrol1006.eqiad.wmnet with OS bullseye [production]
07:46 <taavi@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "cloudcontrol1006 - taavi@cumin1001" [production]
07:45 <taavi@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "cloudcontrol1006 - taavi@cumin1001" [production]
07:37 <joal@deploy2002> helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply [production]
07:37 <joal@deploy2002> helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply [production]
07:36 <taavi@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:36 <taavi@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: assign new IPs to cloudcontrol1006 - taavi@cumin1001" [production]
07:35 <taavi@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: assign new IPs to cloudcontrol1006 - taavi@cumin1001" [production]
07:32 <taavi@cumin1001> START - Cookbook sre.dns.netbox [production]
07:31 <taavi@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:31 <taavi@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: assign new IPs to cloudcontrol1006 - taavi@cumin1001" [production]
07:30 <taavi@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: assign new IPs to cloudcontrol1006 - taavi@cumin1001" [production]
07:28 <taavi@cumin1001> START - Cookbook sre.dns.netbox [production]
05:11 <isaranto@deploy2002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
05:10 <isaranto@deploy2002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
2023-10-01 §
01:38 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2172 (T343198)', diff saved to https://phabricator.wikimedia.org/P52799 and previous config saved to /var/cache/conftool/dbconfig/20231001-013851-arnaudb.json [production]
01:23 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P52798 and previous config saved to /var/cache/conftool/dbconfig/20231001-012344-arnaudb.json [production]
01:08 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P52797 and previous config saved to /var/cache/conftool/dbconfig/20231001-010838-arnaudb.json [production]
00:53 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2172 (T343198)', diff saved to https://phabricator.wikimedia.org/P52796 and previous config saved to /var/cache/conftool/dbconfig/20231001-005332-arnaudb.json [production]
2023-09-30 §
19:44 <arnaudb@cumin1001> dbctl commit (dc=all): 'Depooling db2172 (T343198)', diff saved to https://phabricator.wikimedia.org/P52795 and previous config saved to /var/cache/conftool/dbconfig/20230930-194448-arnaudb.json [production]
19:44 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance [production]