|
2020-02-26
ยง
|
| 15:47 |
<wm-bot> |
<root> Migrated to 2020 Kubernetes cluster |
[tools.noclaims] |
| 15:46 |
<wm-bot> |
<root> Migrated to 2020 Kubernetes cluster |
[tools.nli-wiki] |
| 15:46 |
<wm-bot> |
<root> Migrated to 2020 Kubernetes cluster |
[tools.niosh] |
| 15:46 |
<wm-bot> |
<root> Migrated to 2020 Kubernetes cluster |
[tools.neechal] |
| 15:46 |
<wm-bot> |
<root> Running k8s-2020-migrate.sh with prefix=n |
[tools.admin] |
| 15:25 |
<addshore> |
addshore@mwmaint1002:~$ time mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --batch-size=50 --sleep=1 --file=20to30holes-25feb2229 # T219123 |
[production] |
| 15:19 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 15:17 |
<volans@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 15:06 |
<ottomata> |
dropped and re-added backfilled partitions on event.CentralNoticeImpression table to propogate schema alter on main table - T244771 |
[analytics] |
| 14:54 |
<volans@cumin2001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) |
[production] |
| 14:54 |
<volans@cumin2001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 14:51 |
<volans@cumin2001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 14:46 |
<volans@cumin2001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 14:26 |
<kalle_> |
Deploy latest from Git master: 20d9123, d8447da (T206485), f1f1143, 07e603c, cbfb487 (T179229), 0a7a488, bcd8507, 3225fe2, ebbad49 (T234597), b831a16, 879bc31, 52931bc, 24aa5aa, 93c9f0b, 90df124 (T243384), 1dbe811, 8a3cc99 (T244345), 624274d (T167300, T243376) |
[wikispeech] |
| 14:19 |
<sebastian-wmse> |
Deploy latest from Git master: 13787aa (T192683), fc44dd4, 15214e8 (T192683) |
[wikispeech] |
| 14:19 |
<volans@cumin2001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 14:19 |
<volans@cumin2001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 14:12 |
<volans@cumin2001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 14:11 |
<volans@cumin2001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 14:05 |
<gehel> |
restart of elasticsearch on cloudelastic for JVM upgrade completed |
[production] |
| 14:03 |
<XioNoX> |
deactivate BGP to AS23930 on cr1-eqsin, will re-enable when their technical issues are fixed and they notify us |
[production] |
| 14:00 |
<elukey> |
run apt-get clean on notebook1004 to free some space - T224682 |
[production] |
| 13:45 |
<XioNoX> |
ganeti2001:~$ sudo gnt-instance shutdown apt2001.wikimedia.org - T224576 |
[production] |
| 12:26 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:26 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:24 |
<kartik@deploy1001> |
Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit|416973|ContentTranslation: Set cookieDomain for Production]] (duration: 01m 04s) |
[production] |
| 12:11 |
<kartik@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit|574469|Enable CX out of beta in eu, sw, and ta Wikipedias (T245446, T245447, T245448)]] take II (duration: 01m 05s) |
[production] |
| 12:10 |
<kartik@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit|574469|Enable CX out of beta in eu, sw, and ta Wikipedias (T245446, T245447, T245448)]] (duration: 01m 15s) |
[production] |
| 12:05 |
<volans> |
uploaded spicerack_0.0.31-1_amd64.deb to apt.wikimedia.org stretch-wikimedia |
[production] |
| 11:45 |
<jbond42> |
changing uid/gid of reprepro effects release[12]001/install[12]002 |
[production] |
| 11:05 |
<moritzm> |
rolling out remaining PHP 7.0 security updates |
[production] |
| 10:57 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) |
[production] |
| 10:52 |
<moritzm> |
installing clamav security updates on mendelevium (ticket.wikimedia.org |
[production] |
| 10:03 |
<elukey> |
upgrade prometheus-mcrouter-exporter 0.1.0+git20200225-1 to all cumin alias parsoid/deployment-servers/mw-maintenance |
[production] |
| 09:54 |
<elukey> |
upgrade prometheus-mcrouter-exporter 0.1.0+git20200225-1 to all cumin alias all-mw-eqiad |
[production] |
| 09:50 |
<joal> |
Force delete old api/cirrus events from HDFS trash to free some space |
[analytics] |
| 09:37 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.roll-restart-workers |
[production] |
| 09:34 |
<elukey> |
roll restart the Hadoop Analytcs workers for openjdk upgrades |
[production] |
| 09:32 |
<elukey> |
upgrade prometheus-mcrouter-exporter 0.1.0+git20200225-1 to all cumin alias all-mw-codfw |
[production] |
| 09:18 |
<gehel> |
restarting elasticsearch on cloudelastic for JVM upgrade |
[production] |
| 08:51 |
<elukey> |
upload prometheus-mcrouter-exporter 0.1.0+git20200225-1 to stretch-wikimedia |
[production] |
| 08:38 |
<elukey> |
upgrade prometheus-mcrouter-exporter on mwdebug1001 to test the new version |
[production] |
| 06:19 |
<marostegui> |
Stop MySQL and poweroff db1084 for BBU replacement - T245647 |
[production] |
| 06:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool es1019 after on-site maintenance T243963', diff saved to https://phabricator.wikimedia.org/P10530 and previous config saved to /var/cache/conftool/dbconfig/20200226-061710-marostegui.json |
[production] |
| 06:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Restore es1017 (master) original weight (0) T243963', diff saved to https://phabricator.wikimedia.org/P10529 and previous config saved to /var/cache/conftool/dbconfig/20200226-061640-marostegui.json |
[production] |
| 06:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1084 for BBU replacement - T245647', diff saved to https://phabricator.wikimedia.org/P10528 and previous config saved to /var/cache/conftool/dbconfig/20200226-060906-marostegui.json |
[production] |
| 05:41 |
<kart_> |
Updated cxserver to 2020-02-24-110149-production (T227183) |
[production] |
| 05:35 |
<kartik@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 05:31 |
<kartik@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'cxserver' for release 'production' . |
[production] |
| 05:29 |
<kartik@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . |
[production] |