2351-2400 of 10000 results (38ms)
2021-03-11 §
09:19 <effie> upgrade memcached on mc1020, mc2020 [production]
09:16 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be1036.eqiad.wmnet [production]
09:11 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1035.eqiad.wmnet [production]
09:08 <akosiaris@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . [production]
09:04 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be1035.eqiad.wmnet [production]
09:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1174', diff saved to https://phabricator.wikimedia.org/P14767 and previous config saved to /var/cache/conftool/dbconfig/20210311-090342-marostegui.json [production]
09:03 <marostegui@cumin1001> dbctl commit (dc=all): 'db1079 (re)pooling @ 100%: Repool db1079 after schema change', diff saved to https://phabricator.wikimedia.org/P14766 and previous config saved to /var/cache/conftool/dbconfig/20210311-090312-root.json [production]
09:03 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1033.eqiad.wmnet [production]
08:58 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be1033.eqiad.wmnet [production]
08:58 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1032.eqiad.wmnet [production]
08:51 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be1032.eqiad.wmnet [production]
08:50 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1031.eqiad.wmnet [production]
08:48 <marostegui@cumin1001> dbctl commit (dc=all): 'db1079 (re)pooling @ 60%: Repool db1079 after schema change', diff saved to https://phabricator.wikimedia.org/P14765 and previous config saved to /var/cache/conftool/dbconfig/20210311-084809-root.json [production]
08:43 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be1031.eqiad.wmnet [production]
08:41 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1030.eqiad.wmnet [production]
08:34 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be1030.eqiad.wmnet [production]
08:33 <marostegui@cumin1001> dbctl commit (dc=all): 'db1079 (re)pooling @ 30%: Repool db1079 after schema change', diff saved to https://phabricator.wikimedia.org/P14764 and previous config saved to /var/cache/conftool/dbconfig/20210311-083305-root.json [production]
08:30 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1029.eqiad.wmnet [production]
08:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2109', diff saved to https://phabricator.wikimedia.org/P14762 and previous config saved to /var/cache/conftool/dbconfig/20210311-082546-marostegui.json [production]
08:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2074', diff saved to https://phabricator.wikimedia.org/P14761 and previous config saved to /var/cache/conftool/dbconfig/20210311-082528-marostegui.json [production]
08:24 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2074', diff saved to https://phabricator.wikimedia.org/P14760 and previous config saved to /var/cache/conftool/dbconfig/20210311-082445-marostegui.json [production]
08:24 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be1029.eqiad.wmnet [production]
08:22 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1028.eqiad.wmnet [production]
08:18 <marostegui@cumin1001> dbctl commit (dc=all): 'db1079 (re)pooling @ 10%: Repool db1079 after schema change', diff saved to https://phabricator.wikimedia.org/P14759 and previous config saved to /var/cache/conftool/dbconfig/20210311-081801-root.json [production]
08:16 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ms-be1028.eqiad.wmnet [production]
08:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2108 T275633', diff saved to https://phabricator.wikimedia.org/P14758 and previous config saved to /var/cache/conftool/dbconfig/20210311-081010-marostegui.json [production]
08:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db2148 to s2 T275633', diff saved to https://phabricator.wikimedia.org/P14757 and previous config saved to /var/cache/conftool/dbconfig/20210311-080944-marostegui.json [production]
07:43 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1079', diff saved to https://phabricator.wikimedia.org/P14756 and previous config saved to /var/cache/conftool/dbconfig/20210311-074352-marostegui.json [production]
07:37 <marostegui@cumin1001> dbctl commit (dc=all): 'db1136 (re)pooling @ 100%: Repool db1136 after schema change', diff saved to https://phabricator.wikimedia.org/P14755 and previous config saved to /var/cache/conftool/dbconfig/20210311-073741-root.json [production]
07:22 <marostegui@cumin1001> dbctl commit (dc=all): 'db1136 (re)pooling @ 60%: Repool db1136 after schema change', diff saved to https://phabricator.wikimedia.org/P14754 and previous config saved to /var/cache/conftool/dbconfig/20210311-072237-root.json [production]
07:07 <marostegui@cumin1001> dbctl commit (dc=all): 'db1136 (re)pooling @ 30%: Repool db1136 after schema change', diff saved to https://phabricator.wikimedia.org/P14753 and previous config saved to /var/cache/conftool/dbconfig/20210311-070734-root.json [production]
06:52 <marostegui@cumin1001> dbctl commit (dc=all): 'db1136 (re)pooling @ 10%: Repool db1136 after schema change', diff saved to https://phabricator.wikimedia.org/P14752 and previous config saved to /var/cache/conftool/dbconfig/20210311-065230-root.json [production]
06:48 <marostegui> Stop mysql on db2108 to clone db2148 T275633 [production]
06:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2108 T275633', diff saved to https://phabricator.wikimedia.org/P14750 and previous config saved to /var/cache/conftool/dbconfig/20210311-064821-marostegui.json [production]
06:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1136', diff saved to https://phabricator.wikimedia.org/P14749 and previous config saved to /var/cache/conftool/dbconfig/20210311-063814-marostegui.json [production]
06:36 <marostegui> Drop testreduce from m5 - T276787 [production]
05:34 <thcipriani> restarted apache2 on gerrit1001 [production]
00:50 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2219.codfw.wmnet [production]
00:49 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2218.codfw.wmnet [production]
00:49 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2217.codfw.wmnet [production]
00:48 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2216.codfw.wmnet [production]
00:48 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2215.codfw.wmnet [production]
2021-03-10 §
23:49 <mholloway-shell@deploy1002> Synchronized php-1.36.0-wmf.34/extensions/EventLogging: EventLogging: Stream always in sample if the user is in debugMode (T276515) (duration: 01m 23s) [production]
23:41 <dwisehaupt> disabled silverpop daily run in process-control until utf8mb4 conversion completes on frdev1001 [production]
23:12 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1004.eqiad.wmnet with reason: REIMAGE [production]
23:10 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1004.eqiad.wmnet with reason: REIMAGE [production]
23:10 <legoktm@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts registry1002.eqiad.wmnet [production]
23:01 <legoktm@cumin1001> START - Cookbook sre.hosts.decommission for hosts registry1002.eqiad.wmnet [production]
22:55 <legoktm@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts registry[2001-2002].codfw.wmnet [production]
22:51 <andrewbogott> updating puppet compiler facts to catch up with a new custom fact [production]