2101-2150 of 10000 results (77ms)
2022-11-03 §
09:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P37893 and previous config saved to /var/cache/conftool/dbconfig/20221103-093901-marostegui.json [production]
09:36 <jmm@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti1025.eqiad.wmnet'] [production]
09:26 <elukey@cumin1001> END (PASS) - Cookbook sre.ores.roll-restart-workers (exit_code=0) for ORES codfw cluster: Roll restart of ORES's daemons. [production]
09:23 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1136', diff saved to https://phabricator.wikimedia.org/P37892 and previous config saved to /var/cache/conftool/dbconfig/20221103-092353-marostegui.json [production]
09:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1136 (T321123)', diff saved to https://phabricator.wikimedia.org/P37891 and previous config saved to /var/cache/conftool/dbconfig/20221103-090844-marostegui.json [production]
09:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1136 (T321123)', diff saved to https://phabricator.wikimedia.org/P37890 and previous config saved to /var/cache/conftool/dbconfig/20221103-090631-marostegui.json [production]
09:06 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1136.eqiad.wmnet with reason: Maintenance [production]
09:06 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1136.eqiad.wmnet with reason: Maintenance [production]
09:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1127 (T321123)', diff saved to https://phabricator.wikimedia.org/P37889 and previous config saved to /var/cache/conftool/dbconfig/20221103-090607-marostegui.json [production]
09:05 <elukey@cumin1001> START - Cookbook sre.ores.roll-restart-workers for ORES codfw cluster: Roll restart of ORES's daemons. [production]
09:02 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host ganeti1025.eqiad.wmnet with OS bullseye [production]
09:02 <jmm@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1025.eqiad.wmnet with OS bullseye [production]
08:56 <elukey@cumin1001> END (PASS) - Cookbook sre.ores.roll-restart-workers (exit_code=0) for ORES eqiad cluster: Roll restart of ORES's daemons. [production]
08:53 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host ganeti1025.eqiad.wmnet with OS bullseye [production]
08:53 <jmm@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti1025.eqiad.wmnet with OS bullseye [production]
08:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P37888 and previous config saved to /var/cache/conftool/dbconfig/20221103-085059-marostegui.json [production]
08:44 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host ganeti1025.eqiad.wmnet with OS bullseye [production]
08:43 <jmm@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ganeti1025.eqiad.wmnet with OS bullseye [production]
08:39 <moritzm> installing ruby-nokogiri security updates [production]
08:37 <elukey@cumin1001> START - Cookbook sre.ores.roll-restart-workers for ORES eqiad cluster: Roll restart of ORES's daemons. [production]
08:35 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P37887 and previous config saved to /var/cache/conftool/dbconfig/20221103-083549-marostegui.json [production]
08:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1127 (T321123)', diff saved to https://phabricator.wikimedia.org/P37886 and previous config saved to /var/cache/conftool/dbconfig/20221103-082040-marostegui.json [production]
08:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1127 (T321123)', diff saved to https://phabricator.wikimedia.org/P37885 and previous config saved to /var/cache/conftool/dbconfig/20221103-081827-marostegui.json [production]
08:18 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1127.eqiad.wmnet with reason: Maintenance [production]
08:18 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1127.eqiad.wmnet with reason: Maintenance [production]
08:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T321123)', diff saved to https://phabricator.wikimedia.org/P37884 and previous config saved to /var/cache/conftool/dbconfig/20221103-081805-marostegui.json [production]
08:17 <moritzm> installing glibc security updates on buster [production]
08:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P37883 and previous config saved to /var/cache/conftool/dbconfig/20221103-080257-marostegui.json [production]
08:01 <moritzm> installing exim4 security updates [production]
07:58 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host ganeti1025.eqiad.wmnet with OS bullseye [production]
07:55 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1025.eqiad.wmnet with reason: Remove from cluster for eventual reimage [production]
07:55 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1025.eqiad.wmnet with reason: Remove from cluster for eventual reimage [production]
07:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P37882 and previous config saved to /var/cache/conftool/dbconfig/20221103-074748-marostegui.json [production]
07:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T321123)', diff saved to https://phabricator.wikimedia.org/P37881 and previous config saved to /var/cache/conftool/dbconfig/20221103-073240-marostegui.json [production]
07:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1101:3317 (T321123)', diff saved to https://phabricator.wikimedia.org/P37880 and previous config saved to /var/cache/conftool/dbconfig/20221103-073028-marostegui.json [production]
07:30 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1101.eqiad.wmnet with reason: Maintenance [production]
07:30 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1101.eqiad.wmnet with reason: Maintenance [production]
07:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T321123)', diff saved to https://phabricator.wikimedia.org/P37879 and previous config saved to /var/cache/conftool/dbconfig/20221103-073004-marostegui.json [production]
07:14 <marostegui> Create idm and idm_staging databases on m5 T320426 [production]
07:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P37878 and previous config saved to /var/cache/conftool/dbconfig/20221103-071455-marostegui.json [production]
06:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P37877 and previous config saved to /var/cache/conftool/dbconfig/20221103-065946-marostegui.json [production]
06:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T321123)', diff saved to https://phabricator.wikimedia.org/P37876 and previous config saved to /var/cache/conftool/dbconfig/20221103-064438-marostegui.json [production]
06:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1098:3317 (T321123)', diff saved to https://phabricator.wikimedia.org/P37875 and previous config saved to /var/cache/conftool/dbconfig/20221103-064225-marostegui.json [production]
06:42 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1098.eqiad.wmnet with reason: Maintenance [production]
06:42 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1098.eqiad.wmnet with reason: Maintenance [production]
06:40 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1130.eqiad.wmnet with reason: Maintenance [production]
06:40 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1130.eqiad.wmnet with reason: Maintenance [production]
06:39 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2113.codfw.wmnet with reason: Maintenance [production]
06:39 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db2113.codfw.wmnet with reason: Maintenance [production]
2022-11-02 §
23:25 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2177 (T318605)', diff saved to https://phabricator.wikimedia.org/P37874 and previous config saved to /var/cache/conftool/dbconfig/20221102-232540-ladsgroup.json [production]