8751-8800 of 10000 results (95ms)
2022-11-02 §
07:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P37564 and previous config saved to /var/cache/conftool/dbconfig/20221102-070712-marostegui.json [production]
07:06 <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:851604|Deploy GrowthExperiments to 100% users at all wikis but dewiki (T320876)]] (duration: 04m 38s) [production]
07:02 <urbanecm@deploy1002> urbanecm and urbanecm: Backport for [[gerrit:851604|Deploy GrowthExperiments to 100% users at all wikis but dewiki (T320876)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
07:01 <urbanecm@deploy1002> Started scap: Backport for [[gerrit:851604|Deploy GrowthExperiments to 100% users at all wikis but dewiki (T320876)]] [production]
06:59 <marostegui@cumin1001> dbctl commit (dc=all): 'db1143 (re)pooling @ 3%: After upgrade an incident', diff saved to https://phabricator.wikimedia.org/P37563 and previous config saved to /var/cache/conftool/dbconfig/20221102-065903-root.json [production]
06:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P37562 and previous config saved to /var/cache/conftool/dbconfig/20221102-065203-marostegui.json [production]
06:43 <marostegui@cumin1001> dbctl commit (dc=all): 'db1143 (re)pooling @ 1%: After upgrade an incident', diff saved to https://phabricator.wikimedia.org/P37561 and previous config saved to /var/cache/conftool/dbconfig/20221102-064357-root.json [production]
06:36 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37560 and previous config saved to /var/cache/conftool/dbconfig/20221102-063653-marostegui.json [production]
06:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1096:3315 (T321123)', diff saved to https://phabricator.wikimedia.org/P37559 and previous config saved to /var/cache/conftool/dbconfig/20221102-063038-marostegui.json [production]
06:30 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1096.eqiad.wmnet with reason: Maintenance [production]
06:30 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1096.eqiad.wmnet with reason: Maintenance [production]
05:48 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance [production]
05:47 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance [production]
05:47 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1203 (T318605)', diff saved to https://phabricator.wikimedia.org/P37558 and previous config saved to /var/cache/conftool/dbconfig/20221102-054747-ladsgroup.json [production]
05:32 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P37557 and previous config saved to /var/cache/conftool/dbconfig/20221102-053238-ladsgroup.json [production]
05:17 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P37556 and previous config saved to /var/cache/conftool/dbconfig/20221102-051730-ladsgroup.json [production]
05:02 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1203 (T318605)', diff saved to https://phabricator.wikimedia.org/P37555 and previous config saved to /var/cache/conftool/dbconfig/20221102-050222-ladsgroup.json [production]
04:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1203 (T318605)', diff saved to https://phabricator.wikimedia.org/P37554 and previous config saved to /var/cache/conftool/dbconfig/20221102-042904-ladsgroup.json [production]
04:28 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1203.eqiad.wmnet with reason: Maintenance [production]
04:28 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1203.eqiad.wmnet with reason: Maintenance [production]
04:28 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1193 (T318605)', diff saved to https://phabricator.wikimedia.org/P37553 and previous config saved to /var/cache/conftool/dbconfig/20221102-042838-ladsgroup.json [production]
04:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P37552 and previous config saved to /var/cache/conftool/dbconfig/20221102-041330-ladsgroup.json [production]
03:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P37551 and previous config saved to /var/cache/conftool/dbconfig/20221102-035821-ladsgroup.json [production]
03:43 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1193 (T318605)', diff saved to https://phabricator.wikimedia.org/P37550 and previous config saved to /var/cache/conftool/dbconfig/20221102-034312-ladsgroup.json [production]
03:09 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1193 (T318605)', diff saved to https://phabricator.wikimedia.org/P37549 and previous config saved to /var/cache/conftool/dbconfig/20221102-030959-ladsgroup.json [production]
03:09 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1193.eqiad.wmnet with reason: Maintenance [production]
03:09 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1193.eqiad.wmnet with reason: Maintenance [production]
03:09 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1192 (T318605)', diff saved to https://phabricator.wikimedia.org/P37548 and previous config saved to /var/cache/conftool/dbconfig/20221102-030934-ladsgroup.json [production]
02:54 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P37547 and previous config saved to /var/cache/conftool/dbconfig/20221102-025427-ladsgroup.json [production]
02:39 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P37546 and previous config saved to /var/cache/conftool/dbconfig/20221102-023919-ladsgroup.json [production]
02:24 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1192 (T318605)', diff saved to https://phabricator.wikimedia.org/P37545 and previous config saved to /var/cache/conftool/dbconfig/20221102-022408-ladsgroup.json [production]
01:50 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1192 (T318605)', diff saved to https://phabricator.wikimedia.org/P37544 and previous config saved to /var/cache/conftool/dbconfig/20221102-015019-ladsgroup.json [production]
01:50 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance [production]
01:49 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance [production]
01:49 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1178 (T318605)', diff saved to https://phabricator.wikimedia.org/P37543 and previous config saved to /var/cache/conftool/dbconfig/20221102-014955-ladsgroup.json [production]
01:34 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P37542 and previous config saved to /var/cache/conftool/dbconfig/20221102-013447-ladsgroup.json [production]
01:19 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P37541 and previous config saved to /var/cache/conftool/dbconfig/20221102-011937-ladsgroup.json [production]
01:09 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2181 (T318605)', diff saved to https://phabricator.wikimedia.org/P37540 and previous config saved to /var/cache/conftool/dbconfig/20221102-010958-ladsgroup.json [production]
01:04 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1178 (T318605)', diff saved to https://phabricator.wikimedia.org/P37539 and previous config saved to /var/cache/conftool/dbconfig/20221102-010430-ladsgroup.json [production]
00:54 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P37538 and previous config saved to /var/cache/conftool/dbconfig/20221102-005451-ladsgroup.json [production]
00:39 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P37537 and previous config saved to /var/cache/conftool/dbconfig/20221102-003941-ladsgroup.json [production]
00:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1178 (T318605)', diff saved to https://phabricator.wikimedia.org/P37536 and previous config saved to /var/cache/conftool/dbconfig/20221102-002936-ladsgroup.json [production]
00:29 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Maintenance [production]
00:29 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Maintenance [production]
00:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1177 (T318605)', diff saved to https://phabricator.wikimedia.org/P37535 and previous config saved to /var/cache/conftool/dbconfig/20221102-002913-ladsgroup.json [production]
00:24 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2181 (T318605)', diff saved to https://phabricator.wikimedia.org/P37534 and previous config saved to /var/cache/conftool/dbconfig/20221102-002433-ladsgroup.json [production]
00:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P37533 and previous config saved to /var/cache/conftool/dbconfig/20221102-001401-ladsgroup.json [production]
2022-11-01 §
23:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P37532 and previous config saved to /var/cache/conftool/dbconfig/20221101-235853-ladsgroup.json [production]
23:49 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2181 (T318605)', diff saved to https://phabricator.wikimedia.org/P37531 and previous config saved to /var/cache/conftool/dbconfig/20221101-234957-ladsgroup.json [production]
23:49 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance [production]