|
2021-03-09
§
|
| 19:50 |
<marxarelli> |
restoring database dump on deployment-db07 (T276968) |
[releng] |
| 18:49 |
<marxarelli> |
restarting db dump on db06 `mysqldump -h 127.0.0.1 --events --routines --triggers --all-databases -f --single-transaction` (T276968) |
[releng] |
| 18:38 |
<Majavah> |
installing mariadb 10.4 via role::mariadb::beta to db07 T276968 |
[releng] |
| 18:25 |
<marxarelli> |
"View 'labswiki.tag_summary' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them" when using LOCK TABLES" during mysqldump on db06 (T276968) |
[releng] |
| 18:21 |
<Majavah> |
create deployment-db07 as g2.cores8.ram16.disk160 Buster T276968 |
[releng] |
| 18:20 |
<marxarelli> |
disabled puppet on deployment-db06 and started mysqldump (T276968) |
[releng] |
| 18:09 |
<Majavah> |
set deployment-db05 to read-only to avoid issues with T276968 |
[releng] |
| 18:04 |
<marxarelli> |
deleting shut down memc* deployment-prep instances to free up quota for replacement db instances (T276968) |
[releng] |
| 17:25 |
<marxarelli> |
seeing "[ 2886.337845] EXT4-fs error (device vda3): ext4_validate_block_bitmap:" for deployment-db05 |
[releng] |
| 17:22 |
<marxarelli> |
restarting deployment-db05 via horizon |
[releng] |
| 17:22 |
<marxarelli> |
deployment-db05 seems to be acting up (intermittent connection failures) which is causing issues with beta-update-databases-eqiad, which is (possibly) causing post-merge jobs to pile up |
[releng] |
| 16:47 |
<marxarelli> |
still seeing "JobOffer[deployment-deploy01 #3] rejected beta-scap-eqiad: Waiting for next available executor on ‘deployment-deploy01’" despite available executors |
[releng] |
| 16:26 |
<marxarelli> |
builds once again being scheduled on deployment-deploy01 |
[releng] |
| 16:24 |
<marxarelli> |
cycling gearman plugin on integration.wikimedia.org |
[releng] |
| 16:16 |
<marxarelli> |
taking deployment-deploy01 agent offline to mitigate stuck post-merge jobs |
[releng] |
| 13:32 |
<arturo> |
hard-reboot deployment-db05 because issues related to T276922 |
[releng] |
| 12:34 |
<arturo> |
briefly rebooting VM deployment-db05, we need to reboot its hypervisor cloudvirt1038 and failed to migrate to other |
[releng] |
|
2021-03-07
§
|
| 17:46 |
<James_F> |
Deleting deployment-snapshot01, shut off since 2020-10-03. |
[releng] |
| 17:43 |
<James_F> |
Deleting deployment-cumin02, shut off since 2020-10-16. |
[releng] |
| 17:18 |
<Majavah> |
shutdown deployment-memc[04-05] T276707 |
[releng] |
| 16:51 |
<Majavah> |
cherry pick 669436 and 669436 to deployment-puppetmaster04 T276707 |
[releng] |
| 15:52 |
<Majavah> |
redis::shards change shard01 from deployment-memc04 to deployment-memc08, shard02 from deployment-memc05 to deployment-memc10 T276707 |
[releng] |
| 15:44 |
<Majavah> |
create deployment-memc10 on Buster T276707, beta cluster is almost on full quota but will get better when old shutdown Jessie instances will be deleted |
[releng] |
| 15:28 |
<Majavah> |
remove and shard04 (deployment-memc07) from redis::shards, switch shard03 from deployment-memc06 to deployment-memc09, [06-07] are both already shut down and 09 is a new in setup Buster machine to replace it, T276707 T250585 |
[releng] |
| 13:14 |
<Majavah> |
create deployment-memc09 on Buster T276707 |
[releng] |
|
2021-03-05
§
|
| 20:25 |
<James_F> |
Disabling deployment-memc06 on the grounds that it's an unreferenced Jessie box we don't want any more T250585 |
[releng] |
| 20:23 |
<James_F> |
Disabling deployment-memc07 on the grounds that it's an unreferenced Jessie box we don't want any more T250585 |
[releng] |
| 19:36 |
<Majavah> |
release deployment-prep floating ip 185.15.56.7, was used for mailman upgrade which is now on its own project |
[releng] |
| 19:30 |
<Majavah> |
shutdown deployment-etcd-01 to see if anything breaks, will delete if nothing has broken during next week T276462 |
[releng] |
| 19:14 |
<Majavah> |
beta cluster etcd was switched from deployment-etcd-01 to deployment-etcd02 ref T276462 |
[releng] |
| 17:50 |
<Majavah> |
switch deployment-prep hiera key etcd_host to use deployment-etcd02 ref T276462 |
[releng] |
| 13:40 |
<Majavah> |
create deployment-etcd02 and sign its puppet certificate T276462 |
[releng] |
| 13:13 |
<Majavah> |
move profile::etcd::cluster_name hiera key from deployment-etcd prefix to deployment-etcd-01 vm specific |
[releng] |
| 11:48 |
<Majavah> |
live hack beta puppetmaster to fix hopefully trust store location; T276521 and possibly others |
[releng] |
| 08:32 |
<Majavah> |
deployment-logstash03 try to recreate /etc/rsyslog.d using puppet to try to repair T241481, directory is different on deployment-logstash2 |
[releng] |
|
2021-03-04
§
|
| 15:47 |
<hashar> |
Refreshing jobs based on releng/tox-buster to use latest image. That brings in tox installed with python3 instead of python2 # T276384 |
[releng] |
| 15:00 |
<Majavah> |
remove graphoid role from deploymenr-sca[01-02] ref T276102 and it being decomissioned in T242855 |
[releng] |
| 13:18 |
<Majavah> |
shutdown deployment-fluorine02 for a scream test for T276419, I believe everything has been moved to deployment-mwlog01 |
[releng] |
| 12:38 |
<Majavah> |
`git rebase origin/production` on deployment-puppetmaster04 to update few settings for T276419 |
[releng] |
| 12:19 |
<Majavah> |
Beta cluster is now using deployment-mwlog01 instead of deployment-fluorine02 for MediaWiki logs. fluorine02 is still used for some other misc services, these will be migrated soon |
[releng] |
| 12:06 |
<Majavah> |
deployment-prep Delete lists.beta.wmflabs.org DNS record, points to an unassigned floating IP and not used according to Amir |
[releng] |
| 11:02 |
<Majavah> |
live hacking https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/668338/ on deployment-deploy01 to test new deployment-mwlog01 ref T276419 |
[releng] |
| 10:51 |
<Majavah> |
stop bogus service udp2log on deployment-mwlog01, no idea what it is but it was using the same port as udp2log-mw.service is |
[releng] |
| 09:20 |
<hashar> |
Restored analytics/udp2log cause it got to be packaged for Buster # T276422 T180301 |
[releng] |
| 07:47 |
<legoktm> |
rebuilding php*-compile images https://gerrit.wikimedia.org/r/668259 |
[releng] |