|
2021-03-09
§
|
| 21:54 |
<marxarelli> |
restoring from db06 dump on db07 and db08 following `DROP VIEW IF EXISTS user` workaround (T276968) |
[releng] |
| 20:53 |
<marxarelli> |
restore on db07 failed. appears to be a bug w/ mariadb/mysqldump 10.4 compat https://jira.mariadb.org/browse/MDEV-22127 (T276968) |
[releng] |
| 20:53 |
<marxarelli> |
restore on db07 failed. appears to be a bug w/ mariadb/mysqldump 10.4 compat https://jira.mariadb.org/browse/MDEV-22127 |
[releng] |
| 20:39 |
<marxarelli> |
doing `--skip-grant-tables` on deployment-db08 and creating a new root@127.0.0.1 user (T276968) |
[releng] |
| 20:33 |
<Majavah> |
install mariadb on deployment-db08 T276968 |
[releng] |
| 19:59 |
<marxarelli> |
creating new instance deployment-db08 to use as new beta replica db (T276968) |
[releng] |
| 19:56 |
<marxarelli> |
deleting deployment-db05 to free up quota for new replica (T276968) |
[releng] |
| 19:50 |
<marxarelli> |
restoring database dump on deployment-db07 (T276968) |
[releng] |
| 18:49 |
<marxarelli> |
restarting db dump on db06 `mysqldump -h 127.0.0.1 --events --routines --triggers --all-databases -f --single-transaction` (T276968) |
[releng] |
| 18:38 |
<Majavah> |
installing mariadb 10.4 via role::mariadb::beta to db07 T276968 |
[releng] |
| 18:25 |
<marxarelli> |
"View 'labswiki.tag_summary' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them" when using LOCK TABLES" during mysqldump on db06 (T276968) |
[releng] |
| 18:21 |
<Majavah> |
create deployment-db07 as g2.cores8.ram16.disk160 Buster T276968 |
[releng] |
| 18:20 |
<marxarelli> |
disabled puppet on deployment-db06 and started mysqldump (T276968) |
[releng] |
| 18:09 |
<Majavah> |
set deployment-db05 to read-only to avoid issues with T276968 |
[releng] |
| 18:04 |
<marxarelli> |
deleting shut down memc* deployment-prep instances to free up quota for replacement db instances (T276968) |
[releng] |
| 17:25 |
<marxarelli> |
seeing "[ 2886.337845] EXT4-fs error (device vda3): ext4_validate_block_bitmap:" for deployment-db05 |
[releng] |
| 17:22 |
<marxarelli> |
restarting deployment-db05 via horizon |
[releng] |
| 17:22 |
<marxarelli> |
deployment-db05 seems to be acting up (intermittent connection failures) which is causing issues with beta-update-databases-eqiad, which is (possibly) causing post-merge jobs to pile up |
[releng] |
| 16:47 |
<marxarelli> |
still seeing "JobOffer[deployment-deploy01 #3] rejected beta-scap-eqiad: Waiting for next available executor on ‘deployment-deploy01’" despite available executors |
[releng] |
| 16:26 |
<marxarelli> |
builds once again being scheduled on deployment-deploy01 |
[releng] |
| 16:24 |
<marxarelli> |
cycling gearman plugin on integration.wikimedia.org |
[releng] |
| 16:16 |
<marxarelli> |
taking deployment-deploy01 agent offline to mitigate stuck post-merge jobs |
[releng] |
| 13:32 |
<arturo> |
hard-reboot deployment-db05 because issues related to T276922 |
[releng] |
| 12:34 |
<arturo> |
briefly rebooting VM deployment-db05, we need to reboot its hypervisor cloudvirt1038 and failed to migrate to other |
[releng] |
|
2021-03-07
§
|
| 17:46 |
<James_F> |
Deleting deployment-snapshot01, shut off since 2020-10-03. |
[releng] |
| 17:43 |
<James_F> |
Deleting deployment-cumin02, shut off since 2020-10-16. |
[releng] |
| 17:18 |
<Majavah> |
shutdown deployment-memc[04-05] T276707 |
[releng] |
| 16:51 |
<Majavah> |
cherry pick 669436 and 669436 to deployment-puppetmaster04 T276707 |
[releng] |
| 15:52 |
<Majavah> |
redis::shards change shard01 from deployment-memc04 to deployment-memc08, shard02 from deployment-memc05 to deployment-memc10 T276707 |
[releng] |
| 15:44 |
<Majavah> |
create deployment-memc10 on Buster T276707, beta cluster is almost on full quota but will get better when old shutdown Jessie instances will be deleted |
[releng] |
| 15:28 |
<Majavah> |
remove and shard04 (deployment-memc07) from redis::shards, switch shard03 from deployment-memc06 to deployment-memc09, [06-07] are both already shut down and 09 is a new in setup Buster machine to replace it, T276707 T250585 |
[releng] |
| 13:14 |
<Majavah> |
create deployment-memc09 on Buster T276707 |
[releng] |
|
2021-03-05
§
|
| 20:25 |
<James_F> |
Disabling deployment-memc06 on the grounds that it's an unreferenced Jessie box we don't want any more T250585 |
[releng] |
| 20:23 |
<James_F> |
Disabling deployment-memc07 on the grounds that it's an unreferenced Jessie box we don't want any more T250585 |
[releng] |
| 19:36 |
<Majavah> |
release deployment-prep floating ip 185.15.56.7, was used for mailman upgrade which is now on its own project |
[releng] |
| 19:30 |
<Majavah> |
shutdown deployment-etcd-01 to see if anything breaks, will delete if nothing has broken during next week T276462 |
[releng] |
| 19:14 |
<Majavah> |
beta cluster etcd was switched from deployment-etcd-01 to deployment-etcd02 ref T276462 |
[releng] |
| 17:50 |
<Majavah> |
switch deployment-prep hiera key etcd_host to use deployment-etcd02 ref T276462 |
[releng] |
| 13:40 |
<Majavah> |
create deployment-etcd02 and sign its puppet certificate T276462 |
[releng] |
| 13:13 |
<Majavah> |
move profile::etcd::cluster_name hiera key from deployment-etcd prefix to deployment-etcd-01 vm specific |
[releng] |
| 11:48 |
<Majavah> |
live hack beta puppetmaster to fix hopefully trust store location; T276521 and possibly others |
[releng] |
| 08:32 |
<Majavah> |
deployment-logstash03 try to recreate /etc/rsyslog.d using puppet to try to repair T241481, directory is different on deployment-logstash2 |
[releng] |