|
2020-03-18
ยง
|
| 18:11 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/Wikibase.php: SWAT: [[gerrit:580373|Add beta configuration for Wikibase reference formatting (T247416)]] (duration: 01m 07s) |
[production] |
| 18:04 |
<bstorm_> |
removed puppet prefix tools-flannel-etcd T246689 |
[tools] |
| 17:58 |
<bstorm_> |
removed puppet prefix tools-worker T246689 |
[tools] |
| 17:57 |
<bstorm_> |
removed puppet prefix tools-k8s-master T246689 |
[tools] |
| 17:49 |
<bstorm_> |
updated the tools-prometheus security group with the current IP addresses for the prometheus servers |
[clouddb-services] |
| 17:36 |
<bstorm_> |
removed lots of deprecated hiera keys from horizon for the old cluster T246689 |
[tools] |
| 16:59 |
<bstorm_> |
deleting "tools-worker-1002", "tools-worker-1001", "tools-k8s-master-01", "tools-flannel-etcd-03", "tools-k8s-etcd-03", "tools-flannel-etcd-02", "tools-k8s-etcd-02", "tools-flannel-etcd-01", "tools-k8s-etcd-01" T246689 |
[tools] |
| 16:50 |
<James_F> |
Zuul: [mediawiki/services/parsoid] Drop pipeline |
[releng] |
| 16:50 |
<James_F> |
jjb: Dropped parsoid-pipeline-publish and trigger-parsoid-pipeline-publish |
[releng] |
| 16:47 |
<James_F> |
Zuul: Run Math tests for every change in Wikibase T201496 |
[releng] |
| 16:46 |
<wm-bot> |
<zppix1> disable cron jobs |
[tools.zppixbot] |
| 16:43 |
<mutante> |
wtp1025 - Icinga alerted it's running out of disk - 'apt-get clean' lowered disk usage from 97% to 91% |
[production] |
| 16:34 |
<bstorm_> |
hacked in a possible workaround with extra logging T246096 |
[tools.cdnjs] |
| 16:00 |
<hashar@deploy1001> |
Finished scap: testwiki to 1.35.0-wmf.24 and rebuild l10n cache - T233872 (duration: 61m 23s) |
[production] |
| 14:58 |
<hashar@deploy1001> |
Started scap: testwiki to 1.35.0-wmf.24 and rebuild l10n cache - T233872 |
[production] |
| 14:41 |
<vgutierrez> |
disable TLS session tickets in ulsfo - T245616 T170567 |
[production] |
| 14:29 |
<godog> |
add debug to icinga2001 - T247538 |
[production] |
| 14:28 |
<jeh> |
delete cloudvps project T247482 |
[catgraph] |
| 14:28 |
<_joe_> |
restarted php-fpm on mw1283, was throwing SIGILL |
[production] |
| 14:17 |
<marostegui> |
Rename wb_terms on codfw hosts: s8 (wikidatawiki - db2081), s3 (testwikidatawiki - db2109), s4 (commonswiki, testcommonswiki - db2106) T208425 |
[production] |
| 14:06 |
<hashar@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.23 |
[production] |
| 11:59 |
<hashar@deploy1001> |
Synchronized php-1.35.0-wmf.24/includes/objectcache/ObjectCache.php: objectcache: Restore keyspace for LocalServerCache service - T247562 (duration: 01m 07s) |
[production] |
| 11:57 |
<hashar@deploy1001> |
Synchronized php-1.35.0-wmf.23/includes/objectcache/ObjectCache.php: objectcache: Restore keyspace for LocalServerCache service - T247562 (duration: 01m 10s) |
[production] |
| 11:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Decrease db1087, vslow host weight in main, given that the CPU across s8 is now doing a lot better', diff saved to https://phabricator.wikimedia.org/P10715 and previous config saved to /var/cache/conftool/dbconfig/20200318-114259-marostegui.json |
[production] |
| 11:17 |
<ema> |
upload atskafka 0.3 to buster-wikimedia T237993 |
[production] |
| 11:16 |
<kart_> |
EU Mid-day SWAT done |
[production] |
| 11:11 |
<kartik@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit|579893|Enable ContentTranslation as a default tool in Malay, Azerbaijani and Estonian WPs (T246622, T246628, T246629)]], take II (duration: 01m 07s) |
[production] |
| 11:10 |
<kartik@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit|579893|Enable ContentTranslation as a default tool in Malay, Azerbaijani and Estonian WPs (T246622, T246628, T246629)]] (duration: 01m 07s) |
[production] |
| 10:58 |
<_joe_> |
setting num_retries=0 on mw2224 for eventgate-analytics in envoy (T247484) |
[production] |
| 10:58 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:579925|Stop writing to old term store (wb_terms table) in wikidata (T208425)]], take II (duration: 01m 06s) |
[production] |
| 10:55 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:579925|Stop writing to old term store (wb_terms table) in wikidata (T208425)]] (duration: 01m 08s) |
[production] |
| 10:55 |
<arturo> |
[codfw1dev] deleting BGP agent, undoing changes we did for T245606 |
[admin] |
| 10:52 |
<_joe_> |
setting num_retries=0, idle_timeout=5s on mw2223 for eventgate-analytics in envoy (T247484) |
[production] |
| 10:48 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:579925|Stop writing to old term store in testwikidatawiki (T208425)]], take II (duration: 01m 07s) |
[production] |
| 10:45 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:579925|Stop writing to old term store in testwikidatawiki (T208425)]] (duration: 01m 07s) |
[production] |
| 10:33 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:579925|Read from the new term store everywhere (T219123)]], take II (duration: 01m 07s) |
[production] |
| 10:31 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:579925|Read from the new term store everywhere (T219123)]] (duration: 01m 07s) |
[production] |
| 10:14 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:579925|Read from the new term store everywhere (T219123)]], take II (duration: 01m 07s) |
[production] |
| 10:12 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:579925|Read from the new term store everywhere (T219123)]] (duration: 01m 08s) |
[production] |
| 09:43 |
<vgutierrez> |
enabling inbound TLSv1.3 in upload@ulsfo - T170567 |
[production] |
| 09:18 |
<vgutierrez> |
enabling inbound TLSv1.3 in cp4026 - T170567 |
[production] |
| 08:44 |
<marostegui> |
Start replication pc1008 from pc1010 to get some of the new keys so it is not fully empty - T247787 |
[production] |
| 08:14 |
<vgutierrez> |
upgrade ATS to 8.0.6-1wm3 in ulsfo - T170567 |
[production] |
| 07:55 |
<moritzm> |
installing remaining libxslt security updates |
[production] |
| 07:40 |
<oblivian@deploy1001> |
Synchronized wmf-config/ProductionServices.php: eventgate-analytics to use envoy everywhere (duration: 01m 10s) |
[production] |
| 07:08 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:05 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:31 |
<marostegui> |
Reboot pc1008 to try to get its RAID redone - T247787 |
[production] |
| 03:31 |
<Amir1> |
manually creating empty wb_terms |
[releng] |
| 03:16 |
<Amir1> |
dropping wb_terms table from wikidatawiki in beta cluster (T219123 T219175 T208425) |
[releng] |