|
2020-01-31
§
|
| 18:13 |
<bblack> |
restarted ats-tls and varnish-fe on cp4029 |
[production] |
| 18:05 |
<bblack> |
depool varnish-fe on cp4029 |
[production] |
| 18:03 |
<bblack> |
depool ats-tls on cp4029 |
[production] |
| 16:59 |
<marostegui> |
Re-enable notifications on the dbstore1005:3318 check T243871 |
[production] |
| 16:22 |
<bd808> |
Created wikimania-scholarships project and granted qgil and soworu admin rights (T243995) |
[wikimania-scholarships] |
| 14:05 |
<arturo> |
leave tools-prometheus-01 as the backend for tools-prometheus.wmflabs.org for the weekend so grafana dashboards keep working (T238096) |
[tools] |
| 14:00 |
<arturo> |
syncing again prometheus data from tools-prometheus-01 to tools-prometheus-0{3,4} due to some inconsistencies preventing prometheus from starting (T238096) |
[tools] |
| 11:47 |
<Lucas_WMDE> |
lucaswerkmeister-wmde@deployment-deploy01:~$ mwscript createAndPromote.php enwiki 'Lucas Werkmeister (WMDE)' --force --custom-groups=bureaucrat # make myself bureaucrat to make colleagues sysops |
[releng] |
| 09:18 |
<addshore> |
addshore@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildPropertyTerms.php --wiki=wikidatawiki --sleep 4 --batch-size=25 # In a screen for T219301 |
[production] |
| 03:22 |
<mutante> |
powercycling crashed cp3063 |
[production] |
| 01:09 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@322ee4c]: Update mobileapps to 3eec28d (duration: 06m 53s) |
[production] |
| 01:02 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@322ee4c]: Update mobileapps to 3eec28d |
[production] |
| 00:48 |
<James_F> |
Restarting Jenkins. |
[releng] |
| 00:41 |
<mutante> |
contint1001/contint2001 - upgrading jenkins to 2.219 |
[production] |
| 00:36 |
<mutante> |
releases2001: upgrading jenkins to 2.219; install1002: import jenkins 2.219 into jessie-wikimedia APT repo |
[production] |
| 00:31 |
<mutante> |
importing jenkins 2.219 to stretch-wikimedia APT repo; releases1001: upgrading jenkins to 2.219 |
[production] |
|
2020-01-30
§
|
| 21:04 |
<andrewbogott> |
also apt-get install python3-novaclient on tools-prometheus-03 and tools-prometheus-04 to suppress cronspam. Possible real fix for this is https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/569084/ |
[tools] |
| 20:39 |
<andrewbogott> |
apt-get install python3-keystoneclient on tools-prometheus-03 and tools-prometheus-04 to suppress cronspam. Possible real fix for this is https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/569084/ |
[tools] |
| 19:37 |
<mutante> |
copying /var/log/apache2 to /root on all eqiad mw appservers to preserve logs |
[production] |
| 18:07 |
<vgutierrez> |
depool cp4032 and perform a rolling restart of varnish-fe at cp4027-cp4031 - T243634 |
[production] |
| 17:51 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.16/extensions/Wikibase/lib/includes/Store/Sql/Terms/FingerprintableEntityTermStoreTrait.php: wbterms: Fix incorrect deletion of rows in findActuallyUnusedTermIds (T243944) (duration: 01m 06s) |
[production] |
| 17:49 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.16/extensions/Wikibase/repo/maintenance/rebuildItemTerms.php: wbterms: Write only to the new term store in rebuildItemTerms (T243944) (duration: 01m 09s) |
[production] |
| 17:03 |
<vgutierrez> |
repooling cp4032 - T243634 |
[production] |
| 17:02 |
<vgutierrez> |
restarting varnish-frontend on cp4031 before it crashes - T243634 |
[production] |
| 16:27 |
<arturo> |
create VM tools-prometheus-04 as cold standby of tools-prometheus-03 (T238096) |
[tools] |
| 16:26 |
<vgutierrez> |
manually refreshing OCSP stapling response for non-canonical-redirects-3 - T243948 |
[production] |
| 16:25 |
<arturo> |
point tools-prometheus.wmflabs.org proxy to tools-prometheus-03 (T238096) |
[tools] |
| 13:42 |
<arturo> |
disable puppet in prometheus servers while syncing metric data (T238096) |
[tools] |
| 13:14 |
<arturo> |
drop floating IP 185.15.56.60 and FQDN `prometheus.tools.wmcloud.org` because this is not how the prometheus setup is right now. Use a web proxy instead `tools-prometheus-new.wmflabs.org` (T238096) |
[tools] |
| 13:09 |
<arturo> |
created FQDN `prometheus.tools.wmcloud.org` pointing to IPv4 185.15.56.60 (tools-prometheus-03) to test T238096 |
[tools] |
| 12:59 |
<arturo> |
associated floating IPv4 185.15.56.60 to tools-prometheus-03 (T238096) |
[tools] |
| 12:57 |
<arturo> |
created domain `tools.wmcloud.org` in the tools project after some back and forth with designated, permissions and the database. I plan to use this domain to test the new Debian Buster-based prometheus setup (T238096) |
[tools] |
| 12:22 |
<arturo> |
add prometheus 2.7.1+ds-3+k8s+buster to buster-wikimedia T238096 (basically a rebuild from stretch) |
[production] |
| 10:20 |
<arturo> |
create new VM instance tools-prometheus-03 (T238096) |
[tools] |
| 06:23 |
<vgutierrez> |
restarting varnish-frontend on cp4030 before it crashes - T243634 |
[production] |
| 06:21 |
<vgutierrez> |
depool cp4032 - T243634 |
[production] |
| 05:12 |
<vgutierrez> |
restarting varnish-frontend and repooling cp4029 - T243634 |
[production] |
| 05:00 |
<vgutierrez> |
depooling cp4029 |
[production] |
|
2020-01-29
§
|
| 23:55 |
<James_F> |
zuul: [data-values/value-view] Migrate to node10, now that that passes T228453 |
[releng] |
| 23:37 |
<marostegui> |
Remove partitions from db2087:3317 - T239453 |
[production] |
| 20:56 |
<Krinkle> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/568600 |
[releng] |
| 20:07 |
<bd808> |
Created {bastion,login,dev}.toolforge.org service names for Toolforge bastions using Horizon & Designate |
[tools] |
| 18:17 |
<XioNoX> |
move knams netflow sampling to cr3-knams |
[production] |
| 17:47 |
<arturo> |
delete VM arturo-ocata-test no longer in use |
[openstack] |
| 17:19 |
<krinkle@deploy1001> |
Synchronized wmf-config/etcd.php: Ice8dad2 (duration: 01m 10s) |
[production] |
| 14:20 |
<arturo> |
created and deleted a couple of test VMS for https://gerrit.wikimedia.org/r/c/operations/puppet/+/568473 |
[openstack] |
| 01:11 |
<vgutierrez> |
varnish-frontend restarted on cp4031 |
[production] |
| 01:09 |
<vgutierrez> |
repool cp4031 |
[production] |
| 01:05 |
<marostegui> |
Disable notifications for dbstore1005:3318 slave lag - T243871 |
[production] |
| 01:03 |
<vgutierrez> |
depool cp4031 |
[production] |