|
2021-08-16
§
|
| 08:28 |
<gehel> |
repool wdqs eqiad (`confctl --quiet --object-type discovery select 'dnsdisc=wdqs,name=eqiad' set/pooled=true`) - codfw currently overloaded |
[production] |
| 07:47 |
<marostegui> |
Rename aft_feedback tables on db2115, db2131 - T250715 |
[production] |
| 06:41 |
<TimStarling> |
on votewiki, set voter-privacy option to 1 on all prior elections T288924 |
[production] |
| 05:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2088:3312 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17031 and previous config saved to /var/cache/conftool/dbconfig/20210816-055445-root.json |
[production] |
| 05:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2088:3311 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17030 and previous config saved to /var/cache/conftool/dbconfig/20210816-055427-root.json |
[production] |
| 05:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2088:3312 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17029 and previous config saved to /var/cache/conftool/dbconfig/20210816-053941-root.json |
[production] |
| 05:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2088:3311 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17028 and previous config saved to /var/cache/conftool/dbconfig/20210816-053924-root.json |
[production] |
| 05:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2088:3312 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17027 and previous config saved to /var/cache/conftool/dbconfig/20210816-052437-root.json |
[production] |
| 05:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2088:3311 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17026 and previous config saved to /var/cache/conftool/dbconfig/20210816-052420-root.json |
[production] |
| 05:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2088:3312 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17025 and previous config saved to /var/cache/conftool/dbconfig/20210816-050934-root.json |
[production] |
| 05:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2088:3311 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17024 and previous config saved to /var/cache/conftool/dbconfig/20210816-050916-root.json |
[production] |
| 04:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2088:3312 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17023 and previous config saved to /var/cache/conftool/dbconfig/20210816-045430-root.json |
[production] |
| 04:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2088:3311 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17022 and previous config saved to /var/cache/conftool/dbconfig/20210816-045413-root.json |
[production] |
| 04:49 |
<marostegui> |
Upgrade db2088 (s1 and s2) to 10.4.21 |
[production] |
| 04:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2088 (s1 and s2) to upgrade', diff saved to https://phabricator.wikimedia.org/P17021 and previous config saved to /var/cache/conftool/dbconfig/20210816-044906-marostegui.json |
[production] |
|
2021-08-15
§
|
| 20:02 |
<addshore> |
restarting blazegraph on wdqs2004 |
[production] |
| 18:23 |
<wm-bot> |
<lucaswerkmeister> deployed 9235b38189 (Python 3.9, CC T284590) |
[tools.ranker] |
| 18:06 |
<wm-bot> |
<lucaswerkmeister> deployed de504073a8 (style fix) |
[tools.pagepile-visual-filter] |
| 17:54 |
<wm-bot> |
<lucaswerkmeister> deployed 9e864a3b9b (Python 3.9, no issues so far; CC T284590) |
[tools.pagepile-visual-filter] |
| 17:44 |
<James_F> |
Zuul: [mediawiki/extensions/CIForms] Add basic quibble CI |
[releng] |
| 17:30 |
<majavah> |
deploying update jobs-framework-api container list to include bullseye images |
[tools] |
| 17:21 |
<majavah> |
finished initial build of images: php74, jdk17, python39, ruby27 - T284590 |
[tools] |
| 16:51 |
<majavah> |
starting build of initial bullseye based images - T284590 |
[tools] |
| 16:44 |
<majavah> |
tagged and building toollabs-webservice 0.76 with bullseye images defined T284590 |
[tools] |
| 16:13 |
<andrew@deploy1002> |
Finished deploy [horizon/deploy@c23a155]: adding cinder volume resize warning (duration: 03m 52s) |
[production] |
| 16:10 |
<andrew@deploy1002> |
Started deploy [horizon/deploy@c23a155]: adding cinder volume resize warning |
[production] |
| 15:14 |
<majavah> |
building tools-webservice 0.74 (currently live version) to bullseye-tools and bullseye-toolsbeta |
[tools] |
|
2021-08-14
§
|
| 19:21 |
<wm-bot> |
<lucaswerkmeister> installed TemplateStyles extension (turns out it doesn’t do what I wanted to but let’s keep it anyways) |
[tools.notwikilambda] |
| 17:35 |
<majavah> |
add k8s job to rebuild stretch report, with same parameters as now-deactivated grid cron job for jessie |
[tools.os-deprecation] |
| 17:21 |
<bd808> |
Added majavah as co-maintainer and granted git repo access |
[tools.os-deprecation] |
| 15:11 |
<bd808> |
Transferred ownership from [[User:Owner of abandoned tools]] to [[User:Ash Crow]] (T288890) |
[tools.macommune] |
| 15:03 |
<wm-bot> |
<bd808> Updated config for channel renaming that has happened after libera.chat migration |
[tools.stashbot] |
| 14:57 |
<majavah> |
restart after irc disconnect |
[tools.bridgebot] |
| 12:46 |
<wm-bot> |
<lucaswerkmeister> deployed 7a1980f4e2 (l10n updates) |
[tools.lexeme-forms] |
| 03:54 |
<legoktm[m]> |
restarting mailman3 on lists1001, bounce runner crashed (T288880) |
[production] |
|
2021-08-13
§
|
| 20:09 |
<urbanecm> |
Manually start `beta-update-databases-eqiad` CI job |
[releng] |
| 20:06 |
<urbanecm> |
deployment-prep: sudo -u jenkins-deploy /usr/local/bin/wmf-beta-update-databases.py |
[releng] |
| 20:03 |
<urbanecm> |
Kill beta-scap-sync-world job for the usual reason |
[releng] |
| 18:43 |
<bblack> |
reprepro: uploaded gdnsd-3.8.0-1~wmf1 to buster-wikimedia - T252132 |
[production] |
| 17:32 |
<jelto@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on mw[1451-1452,1454-1455].eqiad.wmnet with reason: setup new mediawiki servers in eqiad https://phabricator.wikimedia.org/T279309 |
[production] |
| 17:32 |
<jelto@cumin1001> |
START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on mw[1451-1452,1454-1455].eqiad.wmnet with reason: setup new mediawiki servers in eqiad https://phabricator.wikimedia.org/T279309 |
[production] |
| 17:06 |
<jelto@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw[1451-1452,1454-1455].eqiad.wmnet with reason: setup new mediawiki servers in eqiad https://phabricator.wikimedia.org/T279309 |
[production] |
| 17:05 |
<jelto@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw[1451-1452,1454-1455].eqiad.wmnet with reason: setup new mediawiki servers in eqiad https://phabricator.wikimedia.org/T279309 |
[production] |
| 16:46 |
<elukey> |
cleanup /srv/discovery on stat1007 after https://gerrit.wikimedia.org/r/c/operations/puppet/+/712422 |
[analytics] |
| 15:39 |
<mutante> |
mw1451, mw1452, mw1454 - rebooting after reimage, memcached needs one |
[production] |
| 15:30 |
<mutante> |
mw1453 - racadm serveraction powercycle (down and was working until right before the switch issue) |
[production] |
| 15:18 |
<godog> |
restart pybal on lvs2009, to clear CRITICAL - thanos-swift_443: Servers thanos-fe2002.codfw.wmnet are marked down but pooled |
[production] |
| 15:16 |
<milimetric> |
reran the other three failed jobs successfully |
[analytics] |
| 15:14 |
<godog> |
restart pybal on lvs2010, to clear CRITICAL - thanos-swift_443: Servers thanos-fe2002.codfw.wmnet are marked down but pooled |
[production] |
| 15:02 |
<mutante> |
etherpad1002 - started failed ferm |
[production] |