|
2020-12-04
§
|
| 23:12 |
<James_F> |
Zuul: [mediawiki/extensions/IPInfo] Re-enable phan, move to prod section T266136 |
[releng] |
| 22:33 |
<andrewbogott> |
moving cloudvirt1023 back into the ceph aggregate; it doesn't need upgrades after all T269467 |
[admin] |
| 22:23 |
<andrewbogott> |
moving cloudvirt1023 out of the ceph aggregate and into maintenance for T269467 |
[admin] |
| 22:19 |
<andrewbogott> |
paysing mwoffliner3 for migration do a different hypervisor |
[mwoffliner] |
| 21:24 |
<James_F> |
Docker: Building composer-package images using composer-scratch, hopefully successfully this time. |
[releng] |
| 21:08 |
<James_F> |
Docker: Building composer-package images using composer-scratch |
[releng] |
| 21:06 |
<andrewbogott> |
putting cloudvirt1025 and 1026 back into service because I'm pretty sure they're fixed. T269313 |
[admin] |
| 20:16 |
<James_F> |
Docker: Fix composer-scratch images to use compose v1.10.17. |
[releng] |
| 19:45 |
<James_F> |
Docker: Pushing composer-phpN images that use composer-scratch |
[releng] |
| 19:34 |
<James_F> |
Zuul: Add experimental PHP 8.0 jobs for all the things. T250298 |
[releng] |
| 18:46 |
<James_F> |
Zuul: Added CI for Flink Swift Plugin T262942 |
[releng] |
| 17:22 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~/uploads]$ mwscript importImages.php --wiki=commonswiki --comment-ext=txt --user=Wilfredor . # T269452 |
[production] |
| 15:47 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:45 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:15 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 15:14 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 15:14 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 13:38 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'apertium' for release 'staging' . |
[production] |
| 13:38 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'apertium' for release 'production' . |
[production] |
| 13:07 |
<akosiaris> |
create apertium namespace on k8s clusters. T255672 |
[production] |
| 12:12 |
<arturo> |
manually running `wmcs-purge-backups` again on cloudvirt1024 (T269419) |
[admin] |
| 11:25 |
<arturo> |
icinga downtime cloudvirt1024 for 6 days, to avoid paging noises (T269419) |
[admin] |
| 11:25 |
<arturo> |
last log line referencing cloudvirt1024 is a mistake (T269313) |
[admin] |
| 11:24 |
<arturo> |
icinga downtime cloudvirt1024 for 6 days, to avoid paging noises (T269313) |
[admin] |
| 11:24 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:24 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:17 |
<dcaro> |
Created a new 'standardized' security froup for k8s from ansible toolsbeta-k8s-full-connectivity (T267412) |
[toolsbeta] |
| 10:31 |
<jynus> |
setting db1133 as read-write for backup testing |
[production] |
| 10:28 |
<arturo> |
manually running `wmcs-purge-backups` on cloudvirt1024 (T269419) |
[admin] |
| 10:28 |
<moritzm> |
resetting cumin-check-aliases.service on cumin* hosts |
[production] |
| 10:23 |
<arturo> |
setting expiration to 2020-12-03 to the oldest backy snapshot of every VM in cloudvirt1024 (T269419) |
[admin] |
| 10:12 |
<dcaro> |
Trying to create a whole new etcd member from ansible (T267412) |
[toolsbeta] |
| 09:54 |
<arturo> |
icinga downtime cloudvirt1025 for 6 days (T269313) |
[admin] |
| 09:54 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:54 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:30 |
<moritzm> |
installing zsh security updates on stretch |
[production] |
| 09:26 |
<moritzm> |
installing mutt security updates |
[production] |
| 08:58 |
<moritzm> |
installing lxml security updates |
[production] |
| 07:09 |
<marostegui> |
Stop mysql on clouddb1016 to clone clouddb1020 T267090 |
[production] |
| 07:02 |
<marostegui> |
Increase pvs on db[1151-1155] T269324 T268742 |
[production] |
| 02:16 |
<eileen> |
civicrm revision changed from 913ccdfd2b to 5fa107d32a, config revision is ffe0a99133 |
[production] |
| 01:43 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 01:42 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 01:04 |
<ryankemper> |
T269406 https://grafana.wikimedia.org/d/000000305/maps-performances?viewPanel=11&orgId=1&var-cluster=maps1&from=1606827063027&to=1607043666975 shows that the normal daily dropoff in lag did not occur today, leading to the criticals. It's possible some sort of daily job has failed |
[production] |
| 00:16 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |