251-300 of 10000 results (19ms)
2021-03-09 §
20:53 <marxarelli> restore on db07 failed. appears to be a bug w/ mariadb/mysqldump 10.4 compat https://jira.mariadb.org/browse/MDEV-22127 [releng]
20:39 <marxarelli> doing `--skip-grant-tables` on deployment-db08 and creating a new root@127.0.0.1 user (T276968) [releng]
20:33 <Majavah> install mariadb on deployment-db08 T276968 [releng]
19:59 <marxarelli> creating new instance deployment-db08 to use as new beta replica db (T276968) [releng]
19:56 <marxarelli> deleting deployment-db05 to free up quota for new replica (T276968) [releng]
19:50 <marxarelli> restoring database dump on deployment-db07 (T276968) [releng]
18:49 <marxarelli> restarting db dump on db06 `mysqldump -h 127.0.0.1 --events --routines --triggers --all-databases -f --single-transaction` (T276968) [releng]
18:38 <Majavah> installing mariadb 10.4 via role::mariadb::beta to db07 T276968 [releng]
18:25 <marxarelli> "View 'labswiki.tag_summary' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them" when using LOCK TABLES" during mysqldump on db06 (T276968) [releng]
18:21 <Majavah> create deployment-db07 as g2.cores8.ram16.disk160 Buster T276968 [releng]
18:20 <marxarelli> disabled puppet on deployment-db06 and started mysqldump (T276968) [releng]
18:09 <Majavah> set deployment-db05 to read-only to avoid issues with T276968 [releng]
18:04 <marxarelli> deleting shut down memc* deployment-prep instances to free up quota for replacement db instances (T276968) [releng]
17:25 <marxarelli> seeing "[ 2886.337845] EXT4-fs error (device vda3): ext4_validate_block_bitmap:" for deployment-db05 [releng]
17:22 <marxarelli> restarting deployment-db05 via horizon [releng]
17:22 <marxarelli> deployment-db05 seems to be acting up (intermittent connection failures) which is causing issues with beta-update-databases-eqiad, which is (possibly) causing post-merge jobs to pile up [releng]
16:47 <marxarelli> still seeing "JobOffer[deployment-deploy01 #3] rejected beta-scap-eqiad: Waiting for next available executor on ‘deployment-deploy01’" despite available executors [releng]
16:26 <marxarelli> builds once again being scheduled on deployment-deploy01 [releng]
16:24 <marxarelli> cycling gearman plugin on integration.wikimedia.org [releng]
16:16 <marxarelli> taking deployment-deploy01 agent offline to mitigate stuck post-merge jobs [releng]
13:32 <arturo> hard-reboot deployment-db05 because issues related to T276922 [releng]
12:34 <arturo> briefly rebooting VM deployment-db05, we need to reboot its hypervisor cloudvirt1038 and failed to migrate to other [releng]
2021-03-08 §
21:38 <brennen> Updating dev-images docker-pkg files on primary contint for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/663159 [releng]
08:57 <hashar> Nuked castor cache for labs/striker # T276605 [releng]
2021-03-07 §
17:46 <James_F> Deleting deployment-snapshot01, shut off since 2020-10-03. [releng]
17:43 <James_F> Deleting deployment-cumin02, shut off since 2020-10-16. [releng]
17:18 <Majavah> shutdown deployment-memc[04-05] T276707 [releng]
16:51 <Majavah> cherry pick 669436 and 669436 to deployment-puppetmaster04 T276707 [releng]
15:52 <Majavah> redis::shards change shard01 from deployment-memc04 to deployment-memc08, shard02 from deployment-memc05 to deployment-memc10 T276707 [releng]
15:44 <Majavah> create deployment-memc10 on Buster T276707, beta cluster is almost on full quota but will get better when old shutdown Jessie instances will be deleted [releng]
15:28 <Majavah> remove and shard04 (deployment-memc07) from redis::shards, switch shard03 from deployment-memc06 to deployment-memc09, [06-07] are both already shut down and 09 is a new in setup Buster machine to replace it, T276707 T250585 [releng]
13:14 <Majavah> create deployment-memc09 on Buster T276707 [releng]
2021-03-06 §
19:44 <Majavah> restart deployment-logstash03 to see if it fixes it being empty [releng]
09:48 <Majavah> cherry-pick https://gerrit.wikimedia.org/r/668995 on deployment-puppetmaster04 T276654 [releng]
08:09 <Majavah> deployment-acme-chief change authorized regex for mx to use .eqiad1.wikimedia.cloud domain to fix T276652 [releng]
2021-03-05 §
20:25 <James_F> Disabling deployment-memc06 on the grounds that it's an unreferenced Jessie box we don't want any more T250585 [releng]
20:23 <James_F> Disabling deployment-memc07 on the grounds that it's an unreferenced Jessie box we don't want any more T250585 [releng]
19:36 <Majavah> release deployment-prep floating ip 185.15.56.7, was used for mailman upgrade which is now on its own project [releng]
19:30 <Majavah> shutdown deployment-etcd-01 to see if anything breaks, will delete if nothing has broken during next week T276462 [releng]
19:14 <Majavah> beta cluster etcd was switched from deployment-etcd-01 to deployment-etcd02 ref T276462 [releng]
17:50 <Majavah> switch deployment-prep hiera key etcd_host to use deployment-etcd02 ref T276462 [releng]
13:40 <Majavah> create deployment-etcd02 and sign its puppet certificate T276462 [releng]
13:13 <Majavah> move profile::etcd::cluster_name hiera key from deployment-etcd prefix to deployment-etcd-01 vm specific [releng]
11:48 <Majavah> live hack beta puppetmaster to fix hopefully trust store location; T276521 and possibly others [releng]
08:32 <Majavah> deployment-logstash03 try to recreate /etc/rsyslog.d using puppet to try to repair T241481, directory is different on deployment-logstash2 [releng]
2021-03-04 §
15:47 <hashar> Refreshing jobs based on releng/tox-buster to use latest image. That brings in tox installed with python3 instead of python2 # T276384 [releng]
15:00 <Majavah> remove graphoid role from deploymenr-sca[01-02] ref T276102 and it being decomissioned in T242855 [releng]
13:18 <Majavah> shutdown deployment-fluorine02 for a scream test for T276419, I believe everything has been moved to deployment-mwlog01 [releng]
12:38 <Majavah> `git rebase origin/production` on deployment-puppetmaster04 to update few settings for T276419 [releng]
12:19 <Majavah> Beta cluster is now using deployment-mwlog01 instead of deployment-fluorine02 for MediaWiki logs. fluorine02 is still used for some other misc services, these will be migrated soon [releng]