301-350 of 10000 results (53ms)
2022-07-19 ยง
12:05 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
12:05 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
12:04 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P31407 and previous config saved to /var/cache/conftool/dbconfig/20220719-120416-marostegui.json [production]
12:04 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
12:01 <urbanecm@deploy1002> Synchronized wmf-config/interwiki.php: Update interwiki cache (T310777) (duration: 02m 49s) [production]
12:00 <moritzm> upgrading ganeti/ulsfo to 3.0.2 T312637 [production]
11:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1179 (T312984)', diff saved to https://phabricator.wikimedia.org/P31406 and previous config saved to /var/cache/conftool/dbconfig/20220719-115719-ladsgroup.json [production]
11:52 <urbanecm@deploy1002> Synchronized langlist: Creating blkwiki (T310777) (duration: 02m 42s) [production]
11:49 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Creating blkwiki (T310777) (duration: 02m 35s) [production]
11:49 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P31405 and previous config saved to /var/cache/conftool/dbconfig/20220719-114911-marostegui.json [production]
11:46 <urbanecm@deploy1002> Synchronized wmf-config/logos.php: Creating blkwiki (T310777) (duration: 02m 49s) [production]
11:46 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ml-etcd2001.codfw.wmnet with reason: Switch instance to DRBD, T311686 [production]
11:46 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on ml-etcd2001.codfw.wmnet with reason: Switch instance to DRBD, T311686 [production]
11:43 <urbanecm@deploy1002> Synchronized static/images/project-logos/: Creating blkwiki (T310777) (duration: 02m 56s) [production]
11:42 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P31404 and previous config saved to /var/cache/conftool/dbconfig/20220719-114214-ladsgroup.json [production]
11:41 <urbanecm@deploy1002> rebuilt and synchronized wikiversions files: Creating blkwiki (T310777) [production]
11:37 <urbanecm@deploy1002> Synchronized dblists: Creating blkwiki (T310777) (duration: 02m 52s) [production]
11:34 <urbanecm@deploy1002> Synchronized wmf-config/db-production.php: Creating blkwiki (T310777) (duration: 02m 47s) [production]
11:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1168 (T312990)', diff saved to https://phabricator.wikimedia.org/P31403 and previous config saved to /var/cache/conftool/dbconfig/20220719-113406-marostegui.json [production]
11:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1168 (T312990)', diff saved to https://phabricator.wikimedia.org/P31401 and previous config saved to /var/cache/conftool/dbconfig/20220719-113158-marostegui.json [production]
11:31 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1168.eqiad.wmnet with reason: Maintenance [production]
11:31 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db1168.eqiad.wmnet with reason: Maintenance [production]
11:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1180 (T312990)', diff saved to https://phabricator.wikimedia.org/P31400 and previous config saved to /var/cache/conftool/dbconfig/20220719-113137-marostegui.json [production]
11:27 <moritzm> remove ganeti 3.0.1-2+deb11u0 from buster-wikimedia, superceded by ganeti 3.0.2-1~deb11u1 from Bullseye 11.4 point release T312637 [production]
11:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P31399 and previous config saved to /var/cache/conftool/dbconfig/20220719-112708-ladsgroup.json [production]
11:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P31398 and previous config saved to /var/cache/conftool/dbconfig/20220719-111632-marostegui.json [production]
11:12 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1179 (T312984)', diff saved to https://phabricator.wikimedia.org/P31397 and previous config saved to /var/cache/conftool/dbconfig/20220719-111203-ladsgroup.json [production]
11:08 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
11:08 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ml-etcd2002.codfw.wmnet with reason: Switch instance to plain, T311686 [production]
11:08 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on ml-etcd2002.codfw.wmnet with reason: Switch instance to plain, T311686 [production]
11:07 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
11:07 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
11:06 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
11:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P31396 and previous config saved to /var/cache/conftool/dbconfig/20220719-110127-marostegui.json [production]
11:00 <ayounsi@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts netboxdb2001.codfw.wmnet [production]
11:00 <ayounsi@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
10:59 <moritzm> draining ganeti2020 T310483 [production]
10:56 <ayounsi@cumin1001> START - Cookbook sre.dns.netbox [production]
10:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1180 (T312990)', diff saved to https://phabricator.wikimedia.org/P31395 and previous config saved to /var/cache/conftool/dbconfig/20220719-104622-marostegui.json [production]
10:46 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1179 (T312984)', diff saved to https://phabricator.wikimedia.org/P31394 and previous config saved to /var/cache/conftool/dbconfig/20220719-104559-ladsgroup.json [production]
10:45 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1179.eqiad.wmnet with reason: Maintenance [production]
10:45 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 10:00:00 on db1179.eqiad.wmnet with reason: Maintenance [production]
10:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1180 (T312990)', diff saved to https://phabricator.wikimedia.org/P31393 and previous config saved to /var/cache/conftool/dbconfig/20220719-104414-marostegui.json [production]
10:44 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1180.eqiad.wmnet with reason: Maintenance [production]
10:43 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db1180.eqiad.wmnet with reason: Maintenance [production]
10:43 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1140.eqiad.wmnet with reason: Maintenance [production]
10:43 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db1140.eqiad.wmnet with reason: Maintenance [production]
10:33 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1127', diff saved to https://phabricator.wikimedia.org/P31392 and previous config saved to /var/cache/conftool/dbconfig/20220719-103341-root.json [production]
10:09 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-etcd2002.codfw.wmnet with reason: Switch instance to DRBD, T311686 [production]
10:08 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ml-etcd2002.codfw.wmnet with reason: Switch instance to DRBD, T311686 [production]