|
2021-11-16
ยง
|
| 17:11 |
<pt1979@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:56 |
<jgiannelos@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 16:27 |
<jgiannelos@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 16:23 |
<herron> |
systemctl reset-failed ifup@ens13 on prometheus5001 T273026 |
[production] |
| 16:22 |
<moritzm> |
systemctl reset-failed ifup@esn13 on durum5001 after restart T273026 |
[production] |
| 16:12 |
<jgiannelos@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 16:05 |
<moritzm> |
powercycling ganeti5002 |
[production] |
| 15:53 |
<andrewbogott> |
merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/525220 which makes read-only ldap the default for ldap clients |
[production] |
| 14:44 |
<cmooney@cumin2002> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host rpki2001.codfw.wmnet |
[production] |
| 14:31 |
<cmooney@cumin2002> |
START - Cookbook sre.ganeti.makevm for new host rpki2001.codfw.wmnet |
[production] |
| 14:31 |
<cmooney@cumin2002> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host rpki2002.codfw.wmnet |
[production] |
| 14:24 |
<jynus> |
re-adding backup user to db1108:analytics_meta T284150 |
[production] |
| 14:22 |
<cmooney@cumin2002> |
START - Cookbook sre.ganeti.makevm for new host rpki2002.codfw.wmnet |
[production] |
| 14:18 |
<cmooney@cumin2002> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts rpki2001.codfw.wmnet |
[production] |
| 14:09 |
<cmooney@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts rpki2001.codfw.wmnet |
[production] |
| 13:58 |
<cmooney@cumin2002> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host rpki2001.codfw.wmnet |
[production] |
| 13:51 |
<cmooney@cumin2002> |
START - Cookbook sre.ganeti.makevm for new host rpki2001.codfw.wmnet |
[production] |
| 13:23 |
<moritzm> |
installing debconf bugfix updates on buster |
[production] |
| 13:21 |
<moritzm> |
prune unused packages from ping3001 T295767 |
[production] |
| 13:18 |
<moritzm> |
prune unused packages from ping1001/ping2001 T295767 |
[production] |
| 13:05 |
<moritzm> |
installing psmisc bugfix updates on buster hosts |
[production] |
| 13:04 |
<mmandere@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6007.drmrs.wmnet with OS buster |
[production] |
| 12:45 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 12:41 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 12:29 |
<moritzm> |
installing Linux 4.19.208 updates on buster hosts (no reboots) |
[production] |
| 12:24 |
<mmandere@cumin1001> |
START - Cookbook sre.hosts.reimage for host cp6007.drmrs.wmnet with OS buster |
[production] |
| 12:22 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) for Druid public cluster: Roll restart of Druid jvm daemons. |
[production] |
| 12:13 |
<mmandere@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6006.drmrs.wmnet with OS buster |
[production] |
| 11:55 |
<moritzm> |
failover ganeti master in test cluster to ganeti-test2002 |
[production] |
| 11:34 |
<mmandere@cumin1001> |
START - Cookbook sre.hosts.reimage for host cp6006.drmrs.wmnet with OS buster |
[production] |
| 11:31 |
<btullis@cumin1001> |
START - Cookbook sre.druid.roll-restart-workers for Druid public cluster: Roll restart of Druid jvm daemons. |
[production] |
| 11:03 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.hadoop.roll-restart-masters (exit_code=0) restart masters for Hadoop analytics cluster: Restart of jvm daemons. |
[production] |
| 10:30 |
<mmandere@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6005.drmrs.wmnet with OS buster |
[production] |
| 10:21 |
<ema> |
A:cp re-enable puppet after successful test on cp402[17] T293879 |
[production] |
| 10:20 |
<btullis@cumin1001> |
START - Cookbook sre.hadoop.roll-restart-masters restart masters for Hadoop analytics cluster: Restart of jvm daemons. |
[production] |
| 10:15 |
<moritzm> |
installing testvm2001 |
[production] |
| 10:06 |
<arturo> |
updating deb packages on stretch-wikimedia/thirdparty/kubeadm-k8s-1-21 (T282942) |
[production] |
| 10:02 |
<ema> |
A:cp disable puppet to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/738910 on cp4021 T293879 |
[production] |
| 09:51 |
<mmandere@cumin1001> |
START - Cookbook sre.hosts.reimage for host cp6005.drmrs.wmnet with OS buster |
[production] |
| 09:48 |
<mmandere@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6004.drmrs.wmnet with OS buster |
[production] |
| 09:40 |
<ayounsi@deploy1002> |
Finished deploy [homer/deploy@c570af3]: Homer CR738905 (duration: 01m 25s) |
[production] |
| 09:39 |
<ayounsi@deploy1002> |
Started deploy [homer/deploy@c570af3]: Homer CR738905 |
[production] |
| 09:09 |
<mmandere@cumin1001> |
START - Cookbook sre.hosts.reimage for host cp6004.drmrs.wmnet with OS buster |
[production] |
| 08:54 |
<mmandere@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6003.drmrs.wmnet with OS buster |
[production] |
| 08:14 |
<mmandere@cumin1001> |
START - Cookbook sre.hosts.reimage for host cp6003.drmrs.wmnet with OS buster |
[production] |
| 08:04 |
<mmandere@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6002.drmrs.wmnet with OS buster |
[production] |
| 07:25 |
<mmandere@cumin1001> |
START - Cookbook sre.hosts.reimage for host cp6002.drmrs.wmnet with OS buster |
[production] |
| 02:10 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 02:07 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:28 |
<urbanecm> |
UTC late window done |
[production] |