|
2020-05-15
§
|
| 15:31 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
| 15:27 |
<gehel@cumin2001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 15:23 |
<bstorm_> |
killing the pods to restart the worker |
[tools.refill-api] |
| 14:32 |
<mutante> |
- instance "dannyb" - the above fixed broken puppet run due to missing mysql packages and installed mariadb instead (there was no existing DB in use and apache works just like before, it also showed the default page before my change |
[wildcat] |
| 14:29 |
<mutante> |
- instance "dannyb" - replacing role::simplelamp with role::simplelamp2 |
[wildcat] |
| 14:19 |
<cdanis> |
reverting sysctl net.ipv4.udp_mem to original on netflow3001 |
[production] |
| 14:18 |
<cdanis> |
re-enable puppet on netflow* |
[production] |
| 14:14 |
<cdanis> |
disable puppet on netflow* |
[production] |
| 14:04 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 14:01 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:47 |
<ema> |
cp2029, cp3050: varnish-fe-restart to clear 'child restarted' alerts |
[production] |
| 13:47 |
<vgutierrez> |
downgrade ats to version 8.0.7-1wm7 on cp4032 |
[production] |
| 13:42 |
<vgutierrez> |
upgrade ats to version 8.0.7-1wm8 on cp4032 |
[production] |
| 13:37 |
<mutante> |
rsyncing gerrit git data from gerrit1001 to gerrit1002 (T200739) |
[production] |
| 13:23 |
<elukey> |
roll restart of the Druid analytics cluster to pick up new openjdk + /srv completed |
[analytics] |
| 13:15 |
<elukey> |
turnilo back to druid1001 |
[analytics] |
| 13:13 |
<cdanis> |
increase samplicator recvbuf on netflow3001 & restart samplicator |
[production] |
| 13:03 |
<elukey> |
move turnilo config to druid1002 to ease druid maintenance |
[analytics] |
| 13:01 |
<cdanis> |
increasing sysctl net.ipv4.udp_mem on netflow3001 |
[production] |
| 12:31 |
<elukey> |
move superset config to druid1002 (was druid1003) to ease maintenance |
[analytics] |
| 09:57 |
<vgutierrez> |
upload trafficserver 8.0.7-1wm7 to apt.wm.o (buster) |
[production] |
| 09:21 |
<ema> |
cp2029: attempt forced discard of stuck VCL T236754 |
[production] |
| 09:09 |
<elukey> |
restart druid brokers on druid100[4-6] - locked up due to datasources dropped - T226035 |
[production] |
| 09:08 |
<elukey> |
restart druid brokers on Analytics Public |
[analytics] |
| 08:51 |
<ema> |
cp2029: try out varnish 5.1.3-1wm15 T236754 |
[production] |
| 07:36 |
<XioNoX> |
bumps prefix limit for AS16735 in eqiad |
[production] |
| 05:35 |
<jynus> |
stop replication on pc2009, pc2010 for benchmarking T252761 |
[production] |
| 04:53 |
<volker-e@deploy1001> |
Finished deploy [design/style-guide@dc956a3]: Deploy design/style-guide: (duration: 00m 10s) |
[production] |
| 04:52 |
<volker-e@deploy1001> |
Started deploy [design/style-guide@dc956a3]: Deploy design/style-guide: |
[production] |
| 04:42 |
<vgutierrez> |
repool cp5006 |
[production] |
| 04:28 |
<vgutierrez> |
depool and reboot cp5006 |
[production] |
|
2020-05-14
§
|
| 23:28 |
<bstorm_> |
downtimed cloudvirt1004/6 and cloudvirt-wdqs1003 until tomorrow around this time T252831 |
[admin] |
| 23:24 |
<catrope@deploy1001> |
Synchronized static/images/project-logos/: Revert temporary 20k logo for vecwiki (T252770) (duration: 01m 06s) |
[production] |
| 23:23 |
<RoanKattouw> |
Ran namespaceDupes.php for T252343 |
[production] |
| 23:20 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Create Gapura (Portal) namespace on jvwiki (T252343) (duration: 01m 06s) |
[production] |
| 23:09 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Add *.ub.uni-heidelberg.de and hq.eso.org to $wgCopyUploadDomains (T252600, T252726) (duration: 01m 07s) |
[production] |
| 22:26 |
<Jayprakash12345> |
[Bug] Use validation timestamp instead of proofread timestamp (T252834) |
[tools.indic-wscontest] |
| 22:21 |
<bstorm_> |
upgrading qemu-system-x86 on cloudvirt1006 to backports version T252831 |
[admin] |
| 22:15 |
<bstorm_> |
changing /etc/libvirt/qemu.conf and restarting libvirtd on cloudvirt1006 T252831 |
[admin] |
| 21:43 |
<ryankemper> |
depooled wdqs2006 while lag recovers |
[production] |
| 21:42 |
<ryankemper@cumin1001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
| 21:12 |
<andrewbogott> |
rebuilding cloudvirt1003-wdqs as part of T252831 |
[admin] |
| 21:08 |
<ryankemper@cumin2001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
| 20:16 |
<volans> |
moved codereview.tar.gz and with_r.tar.gz from miscweb1002 to cumin1001 to free space |
[production] |
| 20:15 |
<hashar@deploy1001> |
Synchronized php-1.35.0-wmf.32/skins/Vector/includes/VectorTemplate.php: Allow plain text labels in side bar - T252727 (duration: 01m 06s) |
[production] |
| 19:51 |
<ryankemper@cumin2001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 19:50 |
<ryankemper@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 19:49 |
<ryankemper> |
Depooled wqds1006 in preparation for impending wdqs data xfer |
[production] |
| 18:41 |
<ottomata> |
fixed TLS authentication for Kafka mirror maker on jumbo - T250250 |
[analytics] |
| 18:36 |
<Urbanecm> |
Morning SWAT done |
[production] |