|
2023-02-16
ยง
|
| 18:54 |
<ebysans@deploy1002> |
Finished deploy [analytics/refinery@0f1a930] (thin): Regular analytics weekly train THIN [analytics/refinery@0f1a930] (duration: 00m 07s) |
[production] |
| 18:54 |
<ebysans@deploy1002> |
Started deploy [analytics/refinery@0f1a930] (thin): Regular analytics weekly train THIN [analytics/refinery@0f1a930] |
[production] |
| 18:52 |
<ebysans@deploy1002> |
Finished deploy [analytics/refinery@0f1a930]: Regular analytics weekly train [analytics/refinery@0f1a930] (duration: 07m 11s) |
[production] |
| 18:46 |
<SandraEbele> |
started deploying analytics refinery |
[analytics] |
| 18:45 |
<ebysans@deploy1002> |
Started deploy [analytics/refinery@0f1a930]: Regular analytics weekly train [analytics/refinery@0f1a930] |
[production] |
| 18:37 |
<SandraEbele> |
killed webrequest oozie bundle to deploy refinery changes. |
[production] |
| 18:37 |
<SandraEbele> |
killed webrequest bundle ooze jobs to deploy refinery changes. |
[analytics] |
| 18:28 |
<bd808@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/toolhub: apply |
[production] |
| 18:26 |
<bd808@deploy1002> |
helmfile [eqiad] START helmfile.d/services/toolhub: apply |
[production] |
| 18:25 |
<bd808@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/toolhub: apply |
[production] |
| 18:24 |
<bd808@deploy1002> |
helmfile [codfw] START helmfile.d/services/toolhub: apply |
[production] |
| 18:22 |
<bd808@deploy1002> |
helmfile [staging] DONE helmfile.d/services/toolhub: apply |
[production] |
| 18:21 |
<bd808@deploy1002> |
helmfile [staging] START helmfile.d/services/toolhub: apply |
[production] |
| 18:08 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on db2106.codfw.wmnet with reason: DB crashed T329864 |
[production] |
| 18:08 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 5 days, 0:00:00 on db2106.codfw.wmnet with reason: DB crashed T329864 |
[production] |
| 17:57 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2106.codfw.wmnet with reason: DB crashed T329864 |
[production] |
| 17:57 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on db2106.codfw.wmnet with reason: DB crashed T329864 |
[production] |
| 17:55 |
<dcaro> |
Manually zapped /dev/sdc on cloudcephosd1002, probably a leftover drive since the beginning (or during the reimage the drives changed names, and this one had leftovers from the previous OS) (T329498) |
[admin] |
| 17:47 |
<wm-bot2> |
Added 1 new OSDs ['cloudcephosd1002.eqiad.wmnet'] (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 17:47 |
<wm-bot2> |
Added OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 17:47 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Depool db2106', diff saved to https://phabricator.wikimedia.org/P44678 and previous config saved to /var/cache/conftool/dbconfig/20230216-174704-jynus.json |
[production] |
| 17:42 |
<wm-bot2> |
Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 17:41 |
<wm-bot2> |
Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 17:38 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache-eqiad: JVM upgrades - elukey@cumin1001 |
[production] |
| 17:25 |
<papaul> |
PDU maintenance in rack A8 |
[production] |
| 17:25 |
<papaul> |
PDU maintenance in rack A1 complete |
[production] |
| 17:21 |
<elukey@cumin1001> |
START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-eqiad: JVM upgrades - elukey@cumin1001 |
[production] |
| 17:07 |
<hnowlan@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply |
[production] |
| 17:07 |
<hnowlan@deploy1002> |
helmfile [eqiad] START helmfile.d/services/api-gateway: apply |
[production] |
| 17:06 |
<hnowlan@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/api-gateway: apply |
[production] |
| 17:05 |
<hnowlan@deploy1002> |
helmfile [codfw] START helmfile.d/services/api-gateway: apply |
[production] |
| 16:55 |
<SandraEbele> |
Deployed refinery-source change to remove Github.io from Mediasites definition of referers. |
[analytics] |
| 16:55 |
<SandraEbele> |
Deployed refinery-source change to remove Github.io from Mediasites definition of referees. |
[production] |
| 16:50 |
<wm-bot2> |
Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 16:50 |
<wm-bot2> |
Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 16:21 |
<ladsgroup@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Move EventLogging settings from IS.php to ext-EventLogging.php, part III (T308932) (duration: 06m 54s) |
[production] |
| 16:19 |
<moritzm> |
installing net-snmp security updates on Buster |
[production] |
| 16:11 |
<ladsgroup@deploy1002> |
Synchronized multiversion/MWConfigCacheGenerator.php: Move EventLogging settings from IS.php to ext-EventLogging.php, part II (T308932) (duration: 06m 48s) |
[production] |
| 16:11 |
<hnowlan@deploy1002> |
helmfile [staging] DONE helmfile.d/services/api-gateway: apply |
[production] |
| 16:11 |
<hnowlan@deploy1002> |
helmfile [staging] START helmfile.d/services/api-gateway: apply |
[production] |
| 16:04 |
<ladsgroup@deploy1002> |
Synchronized wmf-config/ext-EventLogging.php: Move EventLogging settings from IS.php to ext-EventLogging.php, part I (T308932) (duration: 07m 05s) |
[production] |
| 16:01 |
<wm-bot2> |
renewed kubeadm certs on toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo |
[toolsbeta] |
| 16:01 |
<wm-bot2> |
Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 16:01 |
<wm-bot2> |
Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 16:00 |
<wm-bot2> |
Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 16:00 |
<wm-bot2> |
Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 15:58 |
<wm-bot2> |
renewed kubeadm certs on toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo |
[toolsbeta] |
| 15:58 |
<wm-bot2> |
Increased quotas by 4000 gigabytes - cookbook ran by fran@wmf3169 |
[tools] |
| 15:55 |
<wm-bot2> |
renewed kubeadm certs on toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo |
[toolsbeta] |
| 15:49 |
<arturo> |
aborrero@cloud-cumin-03:~$ sudo keyholder arm (password in pw) |
[cloudinfra] |