9451-9500 of 10000 results (94ms)
2024-02-26 §
09:00 <jelto@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on etherpad1003.eqiad.wmnet with reason: Upgrade etherpad and switch to bookworm [production]
08:58 <slyngs> IDP switchover to idp2002 [production]
08:51 <XioNoX> deploy "facebookexternalhit" varnish 403 - T358455 [production]
2024-02-25 §
22:47 <Emperor> mute the outbound port utilisation over 80% alert [production]
00:54 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2172 (T357189)', diff saved to https://phabricator.wikimedia.org/P57920 and previous config saved to /var/cache/conftool/dbconfig/20240225-005423-arnaudb.json [production]
00:39 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P57919 and previous config saved to /var/cache/conftool/dbconfig/20240225-003916-arnaudb.json [production]
00:24 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P57918 and previous config saved to /var/cache/conftool/dbconfig/20240225-002410-arnaudb.json [production]
00:09 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2172 (T357189)', diff saved to https://phabricator.wikimedia.org/P57917 and previous config saved to /var/cache/conftool/dbconfig/20240225-000904-arnaudb.json [production]
2024-02-24 §
23:09 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2172 (T357189)', diff saved to https://phabricator.wikimedia.org/P57916 and previous config saved to /var/cache/conftool/dbconfig/20240224-230912-arnaudb.json [production]
23:09 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance [production]
23:08 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance [production]
23:08 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2155 (T357189)', diff saved to https://phabricator.wikimedia.org/P57915 and previous config saved to /var/cache/conftool/dbconfig/20240224-230850-arnaudb.json [production]
22:53 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P57914 and previous config saved to /var/cache/conftool/dbconfig/20240224-225343-arnaudb.json [production]
22:38 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P57913 and previous config saved to /var/cache/conftool/dbconfig/20240224-223837-arnaudb.json [production]
22:23 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2155 (T357189)', diff saved to https://phabricator.wikimedia.org/P57912 and previous config saved to /var/cache/conftool/dbconfig/20240224-222331-arnaudb.json [production]
21:24 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2155 (T357189)', diff saved to https://phabricator.wikimedia.org/P57911 and previous config saved to /var/cache/conftool/dbconfig/20240224-212414-arnaudb.json [production]
21:24 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
21:23 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
21:23 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance [production]
21:23 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance [production]
21:23 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2147 (T357189)', diff saved to https://phabricator.wikimedia.org/P57910 and previous config saved to /var/cache/conftool/dbconfig/20240224-212336-arnaudb.json [production]
21:08 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P57909 and previous config saved to /var/cache/conftool/dbconfig/20240224-210830-arnaudb.json [production]
20:53 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P57908 and previous config saved to /var/cache/conftool/dbconfig/20240224-205323-arnaudb.json [production]
20:38 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2147 (T357189)', diff saved to https://phabricator.wikimedia.org/P57907 and previous config saved to /var/cache/conftool/dbconfig/20240224-203816-arnaudb.json [production]
19:37 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2147 (T357189)', diff saved to https://phabricator.wikimedia.org/P57906 and previous config saved to /var/cache/conftool/dbconfig/20240224-193712-arnaudb.json [production]
19:37 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2147.codfw.wmnet with reason: Maintenance [production]
19:36 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2147.codfw.wmnet with reason: Maintenance [production]
19:36 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2140 (T357189)', diff saved to https://phabricator.wikimedia.org/P57905 and previous config saved to /var/cache/conftool/dbconfig/20240224-193651-arnaudb.json [production]
19:21 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2140', diff saved to https://phabricator.wikimedia.org/P57904 and previous config saved to /var/cache/conftool/dbconfig/20240224-192144-arnaudb.json [production]
19:06 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2140', diff saved to https://phabricator.wikimedia.org/P57903 and previous config saved to /var/cache/conftool/dbconfig/20240224-190638-arnaudb.json [production]
18:51 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2140 (T357189)', diff saved to https://phabricator.wikimedia.org/P57902 and previous config saved to /var/cache/conftool/dbconfig/20240224-185132-arnaudb.json [production]
17:49 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2140 (T357189)', diff saved to https://phabricator.wikimedia.org/P57901 and previous config saved to /var/cache/conftool/dbconfig/20240224-174941-arnaudb.json [production]
17:49 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2140.codfw.wmnet with reason: Maintenance [production]
17:49 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2140.codfw.wmnet with reason: Maintenance [production]
16:56 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2139.codfw.wmnet with reason: Maintenance [production]
16:56 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2139.codfw.wmnet with reason: Maintenance [production]
16:56 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2137 (T357189)', diff saved to https://phabricator.wikimedia.org/P57900 and previous config saved to /var/cache/conftool/dbconfig/20240224-165636-arnaudb.json [production]
16:41 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2137', diff saved to https://phabricator.wikimedia.org/P57899 and previous config saved to /var/cache/conftool/dbconfig/20240224-164129-arnaudb.json [production]
16:26 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2137', diff saved to https://phabricator.wikimedia.org/P57898 and previous config saved to /var/cache/conftool/dbconfig/20240224-162623-arnaudb.json [production]
16:11 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2137 (T357189)', diff saved to https://phabricator.wikimedia.org/P57897 and previous config saved to /var/cache/conftool/dbconfig/20240224-161117-arnaudb.json [production]
15:12 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2137 (T357189)', diff saved to https://phabricator.wikimedia.org/P57896 and previous config saved to /var/cache/conftool/dbconfig/20240224-151234-arnaudb.json [production]
15:12 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: Maintenance [production]
15:12 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: Maintenance [production]
15:12 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2136 (T357189)', diff saved to https://phabricator.wikimedia.org/P57895 and previous config saved to /var/cache/conftool/dbconfig/20240224-151212-arnaudb.json [production]
14:57 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P57894 and previous config saved to /var/cache/conftool/dbconfig/20240224-145706-arnaudb.json [production]
14:42 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P57893 and previous config saved to /var/cache/conftool/dbconfig/20240224-144200-arnaudb.json [production]
14:26 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2136 (T357189)', diff saved to https://phabricator.wikimedia.org/P57892 and previous config saved to /var/cache/conftool/dbconfig/20240224-142653-arnaudb.json [production]
12:47 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2136 (T357189)', diff saved to https://phabricator.wikimedia.org/P57891 and previous config saved to /var/cache/conftool/dbconfig/20240224-124741-arnaudb.json [production]
12:47 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2136.codfw.wmnet with reason: Maintenance [production]
12:47 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2136.codfw.wmnet with reason: Maintenance [production]