|
2020-05-27
§
|
| 06:57 |
<elukey> |
update matomo on stretch-wikimedia to 3.13.5 |
[production] |
| 06:17 |
<elukey> |
superset upgraded to 0.36 |
[analytics] |
| 06:10 |
<elukey@deploy1001> |
Finished deploy [analytics/superset/deploy@369a2dd]: Upgrade Superset to 0.36 - second attempt (duration: 00m 57s) |
[production] |
| 06:09 |
<elukey@deploy1001> |
Started deploy [analytics/superset/deploy@369a2dd]: Upgrade Superset to 0.36 - second attempt |
[production] |
| 06:03 |
<bd808> |
`systemctl start mariadb` on clouddb1001 following reboot (take 2) |
[admin] |
| 05:58 |
<bd808> |
`systemctl start mariadb` on clouddb1001 following reboot |
[admin] |
| 05:53 |
<bd808> |
Hard reboot of clouddb1001 via Horizon. Console unresponsive. |
[admin] |
| 05:52 |
<elukey> |
attempt to upgrade Superset to 0.36 - downtime expected |
[analytics] |
| 05:17 |
<marostegui> |
Remove tmp_3 key from enwiki.recentchanges on db1099:3311 - T206103 |
[production] |
| 04:41 |
<_joe_> |
cassandra cannot start on restbase2009, one of the disk is failed. |
[production] |
| 04:39 |
<_joe_> |
restarting cassandra instances on restbase2009, has a broken disk |
[production] |
| 04:20 |
<marostegui> |
Depool labsdb1011 - T249188 |
[production] |
|
2020-05-26
§
|
| 22:34 |
<bstorm_> |
restored the deployment for maintain-kubeusers so anyone added to the paws.admin group will have admin on the cluster now that the bug is fixed T211096 T246059 |
[paws] |
| 22:05 |
<bstorm_> |
temporarily deleted the deployment for maintain-kubeusers pending patch to fix context creation for new admin accounts T211096 T246059 |
[paws] |
| 22:04 |
<bstorm_> |
created paws-focused PodSecurityPolicies and the prod namespace in the new cluster T211096 |
[paws] |
| 22:03 |
<bstorm_> |
created paws.admin group and kubernetes admin accounts on the new k8s cluster T211096 T246059 |
[paws] |
| 21:34 |
<krinkle@deploy1001> |
Synchronized wmf-config/mc.php: I0fb124b3593 (duration: 01m 05s) |
[production] |
| 21:30 |
<krinkle@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: I2714e2ae26404 (duration: 01m 06s) |
[production] |
| 21:18 |
<krinkle@deploy1001> |
Synchronized wmf-config/profiler.php: Ib0bf8d97b10b, T253674 (duration: 01m 06s) |
[production] |
| 20:29 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.34 refs T253022 |
[production] |
| 20:08 |
<twentyafterfour@deploy1001> |
Finished scap: testwikis wikis to 1.35.0-wmf.34 refs T253022 (duration: 70m 02s) |
[production] |
| 18:58 |
<twentyafterfour@deploy1001> |
Started scap: testwikis wikis to 1.35.0-wmf.34 refs T253022 |
[production] |
| 18:45 |
<bstorm_> |
upgrading maintain-kubeusers to match what is in toolsbeta T246059 T211096 |
[tools] |
| 18:29 |
<bstorm_> |
bootstrapped the new control plane nodes T211096 |
[paws] |
| 18:07 |
<jforrester@deploy1001> |
Pruned MediaWiki: 1.35.0-wmf.30 (duration: 20m 45s) |
[production] |
| 18:02 |
<bblack> |
cr[12]-eqiad: re-route ns0.wikimedia.org to authdns1001 - T241770 |
[production] |
| 18:02 |
<ejegg> |
restarted fundraising jobs: recurring charge, audit processing, deduplication |
[production] |
| 17:57 |
<moritzm> |
installing bind security updates for stretch (only client-side tools/libraries in use) |
[production] |
| 17:47 |
<cdanis> |
netflow3001: disabling puppet and testing some pmacct/librdkafka config tweaks T253128 |
[production] |
| 17:16 |
<James_F> |
1.35.0-wmf.34 was branched at b5012a1e7d0bbd2bf7444b8708d421992bcbe2fb for T253022 |
[production] |
| 16:45 |
<moritzm> |
installing jsp-api bugfix update from Buster point release |
[production] |
| 16:20 |
<bstorm_> |
fix incorrect volume name in kubeadm-config configmap T246122 |
[tools] |
| 16:17 |
<bstorm_> |
fix incorrect volume name in kubeadm-config T246122 |
[toolsbeta] |
| 15:27 |
<bstorm_> |
updated profile::wmcs::kubeadm::kubernetes_version to 1.16.10 for cluster init T211096 |
[paws] |
| 15:22 |
<akosiaris> |
sync kubernetes eqiad namespaces configuration with helmfile |
[production] |
| 15:15 |
<akosiaris> |
sync kubernetes codfw namespaces configuration with helmfile |
[production] |
| 15:08 |
<arturo> |
delete/re-import docker/containerd.io packages in the right version in buster-wikimedia/thirdparty/kubeadm-k8s-1-{15,16} (T250866) |
[production] |
| 15:08 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Add lazy-loading to Wikimedia Foundation powered-by icon T239377 (duration: 00m 57s) |
[production] |
| 15:02 |
<arturo> |
first k8s upgrade failed for yet-to-be-known reasons (T246122) |
[toolsbeta] |
| 15:01 |
<jforrester@deploy1001> |
Synchronized dblists/mobilemainpagelegacy.dblist: Drop enwiki mobile mainpage special casing T32405 (duration: 00m 59s) |
[production] |
| 14:58 |
<mholloway-shell@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 14:57 |
<akosiaris> |
sync staging namespaces configuration |
[production] |
| 14:57 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 14:57 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 14:57 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' . |
[production] |
| 14:56 |
<jforrester@deploy1001> |
Synchronized docroot/noc/: Clear out symlink to mobile.php, now removed (duration: 00m 55s) |
[production] |
| 14:56 |
<mholloway-shell@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 14:54 |
<arturo> |
`aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` (T246122) |
[toolsbeta] |
| 14:54 |
<mholloway-shell@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' . |
[production] |
| 14:54 |
<arturo> |
bump installed version of kubeadm and kubectl to 1.16.10 (T246122) |
[toolsbeta] |