1401-1450 of 10000 results (23ms)
2021-03-09 ยง
16:21 <marostegui@cumin1001> dbctl commit (dc=all): 'db1175 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P14708 and previous config saved to /var/cache/conftool/dbconfig/20210309-162116-root.json [production]
16:06 <marostegui@cumin1001> dbctl commit (dc=all): 'db1175 (re)pooling @ 80%: 10', diff saved to https://phabricator.wikimedia.org/P14707 and previous config saved to /var/cache/conftool/dbconfig/20210309-160613-root.json [production]
15:56 <moritzm> imported prometheus-ircd-exporter 0.2 to apt.wikimedia.org T224579 [production]
15:51 <marostegui@cumin1001> dbctl commit (dc=all): 'db1175 (re)pooling @ 60%: 10', diff saved to https://phabricator.wikimedia.org/P14706 and previous config saved to /var/cache/conftool/dbconfig/20210309-155109-root.json [production]
15:45 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on analytics1072.eqiad.wmnet with reason: REIMAGE [production]
15:43 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on analytics1072.eqiad.wmnet with reason: REIMAGE [production]
15:37 <marostegui@cumin1001> dbctl commit (dc=all): 'db1096:3316 (re)pooling @ 100%: Repooling db1096:3316 after schema change', diff saved to https://phabricator.wikimedia.org/P14705 and previous config saved to /var/cache/conftool/dbconfig/20210309-153715-root.json [production]
15:36 <marostegui@cumin1001> dbctl commit (dc=all): 'db1175 (re)pooling @ 40%: 10', diff saved to https://phabricator.wikimedia.org/P14704 and previous config saved to /var/cache/conftool/dbconfig/20210309-153605-root.json [production]
15:35 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-fe1008.eqiad.wmnet [production]
15:29 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-fe1008.eqiad.wmnet [production]
15:28 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-fe1007.eqiad.wmnet [production]
15:27 <otto@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Declare KaiOS / Inuka event streams - T267344 T267345 T267346 (duration: 00m 58s) [production]
15:22 <marostegui@cumin1001> dbctl commit (dc=all): 'db1096:3316 (re)pooling @ 60%: Repooling db1096:3316 after schema change', diff saved to https://phabricator.wikimedia.org/P14703 and previous config saved to /var/cache/conftool/dbconfig/20210309-152212-root.json [production]
15:21 <marostegui@cumin1001> dbctl commit (dc=all): 'db1175 (re)pooling @ 30%: 10', diff saved to https://phabricator.wikimedia.org/P14702 and previous config saved to /var/cache/conftool/dbconfig/20210309-152102-root.json [production]
15:20 <otto@deploy1002> Synchronized wmf-config/InitialiseSettings.php: WikimediaEvents: Bump session_tick sampling rate to 10% (duration: 00m 58s) [production]
15:18 <elukey> reimage analytics1072 (hadoop hdfs journal node) to buster [production]
15:15 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-fe1007.eqiad.wmnet [production]
15:15 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-fe1006.eqiad.wmnet [production]
15:11 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-fe1006.eqiad.wmnet [production]
15:07 <marostegui@cumin1001> dbctl commit (dc=all): 'db1096:3316 (re)pooling @ 30%: Repooling db1096:3316 after schema change', diff saved to https://phabricator.wikimedia.org/P14701 and previous config saved to /var/cache/conftool/dbconfig/20210309-150708-root.json [production]
15:05 <marostegui@cumin1001> dbctl commit (dc=all): 'db1175 (re)pooling @ 20%: 10', diff saved to https://phabricator.wikimedia.org/P14700 and previous config saved to /var/cache/conftool/dbconfig/20210309-150558-root.json [production]
15:00 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-fe1005.eqiad.wmnet [production]
14:56 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-fe1005.eqiad.wmnet [production]
14:56 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1089.eqiad.wmnet with reason: REIMAGE [production]
14:54 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1090.eqiad.wmnet with reason: REIMAGE [production]
14:53 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1089.eqiad.wmnet with reason: REIMAGE [production]
14:52 <marostegui@cumin1001> dbctl commit (dc=all): 'db1096:3316 (re)pooling @ 10%: Repooling db1096:3316 after schema change', diff saved to https://phabricator.wikimedia.org/P14699 and previous config saved to /var/cache/conftool/dbconfig/20210309-145205-root.json [production]
14:52 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1090.eqiad.wmnet with reason: REIMAGE [production]
14:41 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-fe2008.codfw.wmnet [production]
14:38 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-fe2008.codfw.wmnet [production]
14:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1096:3316 for schema change', diff saved to https://phabricator.wikimedia.org/P14698 and previous config saved to /var/cache/conftool/dbconfig/20210309-143453-marostegui.json [production]
14:32 <volker-e@deploy1002> Finished deploy [design/style-guide@deee49c]: Deploy design/style-guide: deee49c index: Add links to our design process and work guides (#446) (duration: 00m 06s) [production]
14:32 <volker-e@deploy1002> Started deploy [design/style-guide@deee49c]: Deploy design/style-guide: deee49c index: Add links to our design process and work guides (#446) [production]
14:32 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1015.eqiad.wmnet with reason: REIMAGE [production]
14:31 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-fe2007.codfw.wmnet [production]
14:30 <marostegui@cumin1001> dbctl commit (dc=all): 'db1098:3316 (re)pooling @ 100%: Repooling after schema change', diff saved to https://phabricator.wikimedia.org/P14697 and previous config saved to /var/cache/conftool/dbconfig/20210309-143033-root.json [production]
14:30 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1014.eqiad.wmnet with reason: REIMAGE [production]
14:29 <elukey> drain + reimage an-worker1090/89 to Buster [production]
14:29 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1015.eqiad.wmnet with reason: REIMAGE [production]
14:28 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1012.eqiad.wmnet with reason: REIMAGE [production]
14:27 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-fe2007.codfw.wmnet [production]
14:27 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1014.eqiad.wmnet with reason: REIMAGE [production]
14:26 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1012.eqiad.wmnet with reason: REIMAGE [production]
14:25 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-fe2006.codfw.wmnet [production]
14:21 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-fe2006.codfw.wmnet [production]
14:17 <jakob@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'termbox' for release 'production' . [production]
14:15 <marostegui@cumin1001> dbctl commit (dc=all): 'db1098:3316 (re)pooling @ 60%: Repooling after schema change', diff saved to https://phabricator.wikimedia.org/P14696 and previous config saved to /var/cache/conftool/dbconfig/20210309-141529-root.json [production]
14:14 <jakob@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'termbox' for release 'production' . [production]
14:12 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-fe2005.codfw.wmnet [production]
14:11 <jakob@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'test' . [production]