|
2023-02-16
ยง
|
| 18:21 |
<bd808@deploy1002> |
helmfile [staging] START helmfile.d/services/toolhub: apply |
[production] |
| 18:08 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on db2106.codfw.wmnet with reason: DB crashed T329864 |
[production] |
| 18:08 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 5 days, 0:00:00 on db2106.codfw.wmnet with reason: DB crashed T329864 |
[production] |
| 17:57 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2106.codfw.wmnet with reason: DB crashed T329864 |
[production] |
| 17:57 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on db2106.codfw.wmnet with reason: DB crashed T329864 |
[production] |
| 17:55 |
<dcaro> |
Manually zapped /dev/sdc on cloudcephosd1002, probably a leftover drive since the beginning (or during the reimage the drives changed names, and this one had leftovers from the previous OS) (T329498) |
[admin] |
| 17:47 |
<wm-bot2> |
Added 1 new OSDs ['cloudcephosd1002.eqiad.wmnet'] (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 17:47 |
<wm-bot2> |
Added OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 17:47 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'Depool db2106', diff saved to https://phabricator.wikimedia.org/P44678 and previous config saved to /var/cache/conftool/dbconfig/20230216-174704-jynus.json |
[production] |
| 17:42 |
<wm-bot2> |
Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 17:41 |
<wm-bot2> |
Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 17:38 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache-eqiad: JVM upgrades - elukey@cumin1001 |
[production] |
| 17:25 |
<papaul> |
PDU maintenance in rack A8 |
[production] |
| 17:25 |
<papaul> |
PDU maintenance in rack A1 complete |
[production] |
| 17:21 |
<elukey@cumin1001> |
START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-eqiad: JVM upgrades - elukey@cumin1001 |
[production] |
| 17:07 |
<hnowlan@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply |
[production] |
| 17:07 |
<hnowlan@deploy1002> |
helmfile [eqiad] START helmfile.d/services/api-gateway: apply |
[production] |
| 17:06 |
<hnowlan@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/api-gateway: apply |
[production] |
| 17:05 |
<hnowlan@deploy1002> |
helmfile [codfw] START helmfile.d/services/api-gateway: apply |
[production] |
| 16:55 |
<SandraEbele> |
Deployed refinery-source change to remove Github.io from Mediasites definition of referers. |
[analytics] |
| 16:55 |
<SandraEbele> |
Deployed refinery-source change to remove Github.io from Mediasites definition of referees. |
[production] |
| 16:50 |
<wm-bot2> |
Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 16:50 |
<wm-bot2> |
Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 16:21 |
<ladsgroup@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Move EventLogging settings from IS.php to ext-EventLogging.php, part III (T308932) (duration: 06m 54s) |
[production] |
| 16:19 |
<moritzm> |
installing net-snmp security updates on Buster |
[production] |
| 16:11 |
<ladsgroup@deploy1002> |
Synchronized multiversion/MWConfigCacheGenerator.php: Move EventLogging settings from IS.php to ext-EventLogging.php, part II (T308932) (duration: 06m 48s) |
[production] |
| 16:11 |
<hnowlan@deploy1002> |
helmfile [staging] DONE helmfile.d/services/api-gateway: apply |
[production] |
| 16:11 |
<hnowlan@deploy1002> |
helmfile [staging] START helmfile.d/services/api-gateway: apply |
[production] |
| 16:04 |
<ladsgroup@deploy1002> |
Synchronized wmf-config/ext-EventLogging.php: Move EventLogging settings from IS.php to ext-EventLogging.php, part I (T308932) (duration: 07m 05s) |
[production] |
| 16:01 |
<wm-bot2> |
renewed kubeadm certs on toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo |
[toolsbeta] |
| 16:01 |
<wm-bot2> |
Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 16:01 |
<wm-bot2> |
Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 16:00 |
<wm-bot2> |
Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 16:00 |
<wm-bot2> |
Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus |
[admin] |
| 15:58 |
<wm-bot2> |
renewed kubeadm certs on toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo |
[toolsbeta] |
| 15:58 |
<wm-bot2> |
Increased quotas by 4000 gigabytes - cookbook ran by fran@wmf3169 |
[tools] |
| 15:55 |
<wm-bot2> |
renewed kubeadm certs on toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo |
[toolsbeta] |
| 15:49 |
<arturo> |
aborrero@cloud-cumin-03:~$ sudo keyholder arm (password in pw) |
[cloudinfra] |
| 15:40 |
<dcausse@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 15:39 |
<papaul> |
PDU maintenance in rack A1 |
[production] |
| 15:39 |
<dcausse@deploy1002> |
helmfile [eqiad] START helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 15:36 |
<dcausse@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 15:35 |
<dcausse@deploy1002> |
helmfile [codfw] START helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 15:32 |
<dcausse@deploy1002> |
helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 15:30 |
<dcausse@deploy1002> |
helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 15:30 |
<wm-bot2> |
deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/cert-manager (d71994e) - cookbook ran by arturo@nostromo |
[tools] |
| 15:28 |
<wm-bot2> |
deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/cert-manager (d71994e) - cookbook ran by arturo@nostromo |
[toolsbeta] |
| 14:25 |
<taavi@deploy1002> |
Finished scap: Backport for [[gerrit:888664|GrowthExperiments: Enable link recommendation for 6th round wikis (T304550)]] (duration: 09m 23s) |
[production] |
| 14:17 |
<taavi@deploy1002> |
taavi and sgimeno: Backport for [[gerrit:888664|GrowthExperiments: Enable link recommendation for 6th round wikis (T304550)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet |
[production] |
| 14:16 |
<taavi@deploy1002> |
Started scap: Backport for [[gerrit:888664|GrowthExperiments: Enable link recommendation for 6th round wikis (T304550)]] |
[production] |