1451-1500 of 10000 results (55ms)
2022-06-06 §
04:45 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
04:40 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
04:36 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
04:36 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
04:33 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
04:31 <tstarling@deploy1002> Synchronized wmf-config/db-production.php: enable SSL for cross-DC master connections (duration: 03m 10s) [production]
2022-06-05 §
22:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1144:3314 (T298560)', diff saved to https://phabricator.wikimedia.org/P29417 and previous config saved to /var/cache/conftool/dbconfig/20220605-222110-ladsgroup.json [production]
22:21 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance [production]
22:21 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance [production]
22:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298560)', diff saved to https://phabricator.wikimedia.org/P29416 and previous config saved to /var/cache/conftool/dbconfig/20220605-222102-ladsgroup.json [production]
22:05 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P29415 and previous config saved to /var/cache/conftool/dbconfig/20220605-220557-ladsgroup.json [production]
21:50 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P29414 and previous config saved to /var/cache/conftool/dbconfig/20220605-215052-ladsgroup.json [production]
21:35 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298560)', diff saved to https://phabricator.wikimedia.org/P29413 and previous config saved to /var/cache/conftool/dbconfig/20220605-213547-ladsgroup.json [production]
12:53 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1140.eqiad.wmnet with reason: Maintenance [production]
12:53 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1140.eqiad.wmnet with reason: Maintenance [production]
12:53 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1135 (T298560)', diff saved to https://phabricator.wikimedia.org/P29412 and previous config saved to /var/cache/conftool/dbconfig/20220605-125302-ladsgroup.json [production]
12:37 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P29411 and previous config saved to /var/cache/conftool/dbconfig/20220605-123757-ladsgroup.json [production]
12:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1138 (T298560)', diff saved to https://phabricator.wikimedia.org/P29410 and previous config saved to /var/cache/conftool/dbconfig/20220605-122702-ladsgroup.json [production]
12:27 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1138.eqiad.wmnet with reason: Maintenance [production]
12:26 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1138.eqiad.wmnet with reason: Maintenance [production]
12:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298560)', diff saved to https://phabricator.wikimedia.org/P29409 and previous config saved to /var/cache/conftool/dbconfig/20220605-122654-ladsgroup.json [production]
12:22 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P29408 and previous config saved to /var/cache/conftool/dbconfig/20220605-122252-ladsgroup.json [production]
12:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P29407 and previous config saved to /var/cache/conftool/dbconfig/20220605-121149-ladsgroup.json [production]
12:07 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1135 (T298560)', diff saved to https://phabricator.wikimedia.org/P29406 and previous config saved to /var/cache/conftool/dbconfig/20220605-120747-ladsgroup.json [production]
11:56 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P29405 and previous config saved to /var/cache/conftool/dbconfig/20220605-115644-ladsgroup.json [production]
11:41 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298560)', diff saved to https://phabricator.wikimedia.org/P29404 and previous config saved to /var/cache/conftool/dbconfig/20220605-114139-ladsgroup.json [production]
03:58 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host clouddumps1001.wikimedia.org with OS bullseye [production]
03:36 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host clouddumps1001.wikimedia.org with OS bullseye [production]
03:35 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host clouddumps1001.wikimedia.org with OS bullseye [production]
03:35 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host clouddumps1001.wikimedia.org with OS bullseye [production]
02:57 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host clouddumps1001.wikimedia.org with OS bullseye [production]
02:45 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1149 (T298560)', diff saved to https://phabricator.wikimedia.org/P29403 and previous config saved to /var/cache/conftool/dbconfig/20220605-024538-ladsgroup.json [production]
02:45 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1149.eqiad.wmnet with reason: Maintenance [production]
02:45 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1149.eqiad.wmnet with reason: Maintenance [production]
02:45 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298560)', diff saved to https://phabricator.wikimedia.org/P29402 and previous config saved to /var/cache/conftool/dbconfig/20220605-024530-ladsgroup.json [production]
02:30 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P29401 and previous config saved to /var/cache/conftool/dbconfig/20220605-023025-ladsgroup.json [production]
02:15 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P29400 and previous config saved to /var/cache/conftool/dbconfig/20220605-021520-ladsgroup.json [production]
02:00 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298560)', diff saved to https://phabricator.wikimedia.org/P29399 and previous config saved to /var/cache/conftool/dbconfig/20220605-020015-ladsgroup.json [production]
01:37 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host clouddumps1001.wikimedia.org with OS bullseye [production]
01:37 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host clouddumps1001.wikimedia.org with OS bullseye [production]
01:22 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host clouddumps1001.wikimedia.org with OS bullseye [production]
01:08 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host clouddumps1001.wikimedia.org with OS bullseye [production]
00:19 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host clouddumps1001.wikimedia.org with OS bullseye [production]
00:18 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host clouddumps1001.wikimedia.org with OS bullseye [production]
2022-06-04 §
23:51 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host clouddumps1001.wikimedia.org with OS bullseye [production]
23:50 <andrew@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host clouddumps1001.wikimedia.org with OS bullseye [production]
23:32 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host clouddumps1001.wikimedia.org with OS bullseye [production]
23:29 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host clouddumps1001.wikimedia.org with OS bullseye [production]
17:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1148 (T298560)', diff saved to https://phabricator.wikimedia.org/P29398 and previous config saved to /var/cache/conftool/dbconfig/20220604-170633-ladsgroup.json [production]
17:06 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1148.eqiad.wmnet with reason: Maintenance [production]