|
2021-03-19
ยง
|
| 19:39 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw2244.codfw.wmnet |
[production] |
| 19:37 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2252.codfw.wmnet,service=canary |
[production] |
| 19:37 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2251.codfw.wmnet,service=canary |
[production] |
| 19:33 |
<dzahn@cumin1001> |
conftool action : set/weight=1; selector: name=mw2252.codfw.wmnet,service=canary |
[production] |
| 19:33 |
<dzahn@cumin1001> |
conftool action : set/weight=1; selector: name=mw2251.codfw.wmnet,service=canary |
[production] |
| 19:24 |
<mutante> |
deploy2002 - re-enabled puppet, reverted patch of scap-sync-master |
[production] |
| 18:46 |
<mutante> |
deploy2002 - disable puppet, copy modified version of scap-master-sync over it that does not --exclude="**/cache/l10n/*.cdb" (for T275826) |
[production] |
| 17:17 |
<bstorm> |
running `ALTER TABLE account MODIFY COLUMN type ENUM('user','tool','paws');` against the labsdbaccounts database on m5 T276284 |
[admin] |
| 16:01 |
<effie> |
upgrade memcached on mc-gp200* |
[production] |
| 15:44 |
<razzi> |
rebalance kafka partitions for webrequest_text partition 1 |
[analytics] |
| 15:11 |
<dpifke> |
Re-cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/623068 in beta. |
[releng] |
| 14:29 |
<andrewbogott> |
switching admin-monitoring project to use an upstream debian image; I want to see how this affects performance |
[admin] |
| 13:35 |
<addshore> |
reload zuul for https://gerrit.wikimedia.org/r/673472 (https://phabricator.wikimedia.org/T277750) |
[releng] |
| 12:48 |
<Majavah> |
shutdown deployment-sca*, services on them are too old and broken to be useful according to the SREs, have no maintainers and the hosts are running Jessie, T218729 |
[releng] |
| 12:36 |
<klausman@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2002.codfw.wmnet with reason: REIMAGE |
[production] |
| 12:34 |
<klausman@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2002.codfw.wmnet with reason: REIMAGE |
[production] |
| 12:10 |
<effie> |
upgrade memcached on mc1026,mc2026 |
[production] |
| 11:49 |
<Majavah> |
remove now-deleted deployment-cumin02 from hiera allowed cumin masters |
[releng] |
| 11:37 |
<klausman@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. |
[production] |
| 11:37 |
<klausman@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. |
[production] |
| 11:36 |
<klausman@deploy1002> |
helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 11:36 |
<klausman@deploy1002> |
helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 11:30 |
<klausman@deploy1002> |
helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 11:29 |
<klausman@deploy1002> |
helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 11:29 |
<klausman@deploy1002> |
helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 11:29 |
<klausman@deploy1002> |
helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 11:29 |
<klausman@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. |
[production] |
| 11:29 |
<klausman@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. |
[production] |
| 11:27 |
<akosiaris@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. |
[production] |
| 11:27 |
<akosiaris@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. |
[production] |
| 11:20 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2002.codfw.wmnet with reason: REIMAGE |
[production] |
| 11:18 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2002.codfw.wmnet with reason: REIMAGE |
[production] |
| 10:45 |
<klausman@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. |
[production] |
| 10:45 |
<klausman@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. |
[production] |
| 10:42 |
<moritzm> |
installing dbmonitor1002 T224589 |
[production] |
| 10:42 |
<klausman@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. |
[production] |
| 10:42 |
<klausman@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. |
[production] |
| 10:41 |
<klausman@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. |
[production] |
| 10:41 |
<klausman@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. |
[production] |
| 10:11 |
<kharlan@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 10:10 |
<kharlan@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 10:05 |
<kharlan@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 10:04 |
<kharlan@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 09:40 |
<kharlan@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . |
[production] |
| 09:36 |
<jayme@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'proton' for release 'production' . |
[production] |
| 08:54 |
<Majavah> |
remove deployment-restbase02 from cassandra and shut it down T250574 |
[releng] |
| 08:45 |
<Majavah> |
disable puppet and stop restbase service on deployment-restbase02 for T250574 |
[releng] |
| 08:22 |
<elukey> |
upload alluxio 2.4.1 to thirdparty/bigtop15 on stretch/buster-wikimedia |
[production] |
| 07:16 |
<ryankemper> |
T275885 `ryankemper@cumin1001:~$ sudo cumin 'P{relforge*}' 'sudo run-puppet-agent'` (change hadn't been merged when I ran the agent earlier) |
[production] |
| 04:04 |
<eileen> |
civicrm revision changed from 99bf1c9210 to 39d24e8b0a, config revision is 26b02db7ba |
[production] |