|
2020-07-09
§
|
| 11:19 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 11:17 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 11:11 |
<arturo> |
[codfw1dev] rebooting cloudnet2003-dev for testing sysct/puppet behavior (T257552) |
[admin] |
| 11:10 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:10 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:09 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: e6f442c6900524482806aeb1b5162e65bf7c97ac: Enable Quicksurveys for Desktop Improvements Project (T246977) (duration: 01m 06s) |
[production] |
| 11:01 |
<vgutierrez> |
restart ats-tls on cp1085 |
[production] |
| 10:55 |
<_joe_> |
restarting php7.2-fpm on mw1282, workers failing with sigill |
[production] |
| 10:54 |
<_joe_> |
depool mw1282 |
[production] |
| 10:54 |
<mvolz@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'citoid' for release 'production' . |
[production] |
| 10:42 |
<addshore> |
reload zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/610713 T257549 |
[releng] |
| 10:34 |
<mvolz@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'citoid' for release 'production' . |
[production] |
| 10:23 |
<_joe_> |
rolling restart the remaining restbases in eqiad, and all of codfw |
[production] |
| 10:22 |
<mvolz@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'citoid' for release 'staging' . |
[production] |
| 10:09 |
<_joe_> |
restarting restbase on rb1020-22 |
[production] |
| 09:53 |
<_joe_> |
restarting restbase on restbase1024,1023 |
[production] |
| 09:36 |
<_joe_> |
restarting restbase on rb1026,1027 to switch to proton on k8s |
[production] |
| 09:34 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:31 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:28 |
<_joe_> |
restarting restbase on restbase1025 to pick up the switch to k8s of proton |
[production] |
| 09:27 |
<godog> |
bounce thanos-compact on thanos-fe2001 |
[production] |
| 09:25 |
<hashar> |
contint2001: manually pulled all Docker images referenced in jjb. |
[releng] |
| 09:16 |
<arturo> |
manually increasing sysctl value of net.nf_conntrack_max in cloudnet servers (T257552) |
[admin] |
| 09:07 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.change-distro (exit_code=0) |
[production] |
| 08:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1079', diff saved to https://phabricator.wikimedia.org/P11828 and previous config saved to /var/cache/conftool/dbconfig/20200709-085228-marostegui.json |
[production] |
| 08:44 |
<marostegui> |
Stop haproxy on dbproxy1017 before upgrading to buster - T255408 |
[production] |
| 08:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1136', diff saved to https://phabricator.wikimedia.org/P11827 and previous config saved to /var/cache/conftool/dbconfig/20200709-082355-marostegui.json |
[production] |
| 08:23 |
<moritzm> |
imported osm2pgsql 0.96.0+ds-1~bpo9+1 to "main" component T256877 |
[production] |
| 08:22 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.change-distro |
[production] |
| 08:20 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0) |
[production] |
| 08:13 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.stop-cluster |
[production] |
| 08:11 |
<XioNoX> |
disable igmp snooping on msw1-codfw |
[production] |
| 07:59 |
<marostegui> |
Stop db1117:3322 to clone db1084, this will trigger haproxy alerts - T257540 |
[production] |
| 07:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1136', diff saved to https://phabricator.wikimedia.org/P11825 and previous config saved to /var/cache/conftool/dbconfig/20200709-075749-marostegui.json |
[production] |
| 07:19 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:15 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:52 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:49 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 05:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1101:3317', diff saved to https://phabricator.wikimedia.org/P11824 and previous config saved to /var/cache/conftool/dbconfig/20200709-053905-marostegui.json |
[production] |
| 05:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1084 from dbctl', diff saved to https://phabricator.wikimedia.org/P11823 and previous config saved to /var/cache/conftool/dbconfig/20200709-053206-marostegui.json |
[production] |
| 05:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1084', diff saved to https://phabricator.wikimedia.org/P11822 and previous config saved to /var/cache/conftool/dbconfig/20200709-051826-marostegui.json |
[production] |
| 05:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1101:3317', diff saved to https://phabricator.wikimedia.org/P11821 and previous config saved to /var/cache/conftool/dbconfig/20200709-051355-marostegui.json |
[production] |
| 05:11 |
<marostegui> |
Remove revision triggers from db2093:3315 T238966 |
[production] |
| 05:10 |
<marostegui> |
Deploy schema change on s5 codfw, lag will be generated - T238966 |
[production] |
| 04:39 |
<wm-bot> |
<root> Restarted to pick up toolforge.org migration related changes to $HOME/.lighttpd.conf (T257481) |
[tools.refill] |
| 02:55 |
<wm-bot> |
<legoktm> enable redis caching for mostly static data |
[tools.checker] |
| 02:30 |
<wm-bot> |
<root> Hard restart to pick up "replicas: 2" setting in $HOME/service.template |
[tools.checker] |
| 01:43 |
<tzatziki> |
reset email for GseSro |
[production] |
| 01:12 |
<dpifke> |
Cherry picking CL 608973 into beta puppet, should only affect webperf hosts. |
[releng] |
| 00:58 |
<cdanis> |
✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕘🍺 sudo cumin A:cp 'enable-puppet "cdanis deploying I6c1b646e T256395"' |
[production] |