3851-3900 of 10000 results (97ms)
2020-02-18 §
08:23 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1107 after temporary change optimizer options - T245489', diff saved to https://phabricator.wikimedia.org/P10442 and previous config saved to /var/cache/conftool/dbconfig/20200218-082306-marostegui.json [production]
08:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1107 after temporary change optimizer options - T245489', diff saved to https://phabricator.wikimedia.org/P10441 and previous config saved to /var/cache/conftool/dbconfig/20200218-080952-marostegui.json [production]
08:08 <marostegui> Restart MySQL to pick up optimizer_switch changes - T245489 [production]
08:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1107 to temporary change optimizer options - T245489', diff saved to https://phabricator.wikimedia.org/P10440 and previous config saved to /var/cache/conftool/dbconfig/20200218-080623-marostegui.json [production]
07:34 <elukey> powercycle analytics1065 (crashed hours ago, no mgmt console available, no ssh) [production]
06:39 <marostegui> Remove wikiadmin2 from pc1007, pc1008, pc1009 and pc1010 T243512 [production]
06:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Increase weight for db1107 100 -> 200 for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10439 and previous config saved to /var/cache/conftool/dbconfig/20200218-063819-marostegui.json [production]
06:27 <marostegui> Stop haproxy on dbproxy1007 - T245385 [production]
06:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1107 with weight 100 and weight 10 in API for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10438 and previous config saved to /var/cache/conftool/dbconfig/20200218-062459-marostegui.json [production]
06:09 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
06:08 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
2020-02-17 §
19:56 <cdanis> finish enabling TCP-MSS clamping in eqiad [production]
19:49 <cdanis> s/no-op// [production]
19:49 <cdanis> no-op enable TCP-MSS clamping on eqord and eqiad [production]
19:33 <cdanis> no-op enable flowspec change on cr2-eqord and cr2-eqiad [production]
18:25 <elukey> restart kafka on kafka-jumbo1001 to pick up new openjdk updates [production]
17:25 <bblack> GRE MTU mitigations applied to esams cp hosts only - T232602 [production]
15:55 <ayounsi@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
15:50 <ayounsi@cumin1001> START - Cookbook sre.ganeti.makevm [production]
15:48 <ayounsi@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
15:48 <ayounsi@cumin1001> START - Cookbook sre.ganeti.makevm [production]
15:44 <cdanis> ✔️ cdanis@icinga1001.wikimedia.org ~ 🕥☕ sudo systemctl restart ircecho [production]
14:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1107 after 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10422 and previous config saved to /var/cache/conftool/dbconfig/20200217-143146-marostegui.json [production]
14:17 <ema> reprepro includedeb buster-wikimedia ~ema/cadvisor_0.35.0+ds1-4_amd64.deb T183146 [production]
12:34 <XioNoX> add test flowspec rules to cr3-knams [production]
12:34 <moritzm> installing postgresql-9.4 security updates [production]
12:27 <vgutierrez> reboot acmechief instances (kernel upgrade) [production]
10:31 <jynus> dropping all databases from db1140:3313 [production]
10:22 <marostegui@cumin1001> dbctl commit (dc=all): ' db1107 increase API weight from 10 to 15 for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10420 and previous config saved to /var/cache/conftool/dbconfig/20200217-102218-marostegui.json [production]
10:20 <vgutierrez> rolling restart of ats-tls and varnish-fe on ulsfo to enable KA between them - T244464 [production]
10:00 <moritzm> installing Linux 4.9.210 kernels on stretch systems [production]
09:10 <godog> correction, +100G [production]
09:09 <godog> +10G to prometheus/ops fs on prometheus eqiad - T245361 [production]
09:06 <godog> +50G to prometheus/ops fs on prometheus eqiad - T245361 [production]
07:22 <marostegui> Stop haproxy on dbproxy1002 - T245384 [production]
2020-02-15 §
01:01 <cdanis> ✔️ cdanis@an-coord1001.eqiad.wmnet ~ 🕗🍺 sudo systemctl restart hive-server2.service ; sudo systemctl restart hive-metastore.service [production]
2020-02-14 §
23:42 <XenoRyet> updated civicrm from cf86495d44 to 8c77e9e915 [production]
21:01 <volker-e@deploy1001> Finished deploy [design/style-guide@1928c00]: Deploy design/style-guide: (duration: 00m 09s) [production]
21:01 <volker-e@deploy1001> Started deploy [design/style-guide@1928c00]: Deploy design/style-guide: [production]
20:21 <reedy@deploy1001> Synchronized wmf-config/CommonSettings.php: Prevent some logspam T245280 (duration: 01m 05s) [production]
19:27 <XenoRyet> updated civicrm from 55b2afb6eb to cf86495d44 [production]
19:09 <jforrester@deploy1001> Synchronized php-1.35.0-wmf.19/extensions/Wikibase: T245062 Prevent invalid term languages from cached PrefetchingTermLookup (duration: 01m 09s) [production]
17:37 <jforrester@deploy1001> Unlocked for deployment [ALL REPOSITORIES]: Testing T245062 fix on mwdebug1001 (duration: 03m 05s) [production]
17:33 <jforrester@deploy1001> Locking from deployment [ALL REPOSITORIES]: Testing T245062 fix on mwdebug1001 (planned duration: 60m 00s) [production]
16:11 <moritzm> installing git-lfs updates from Buster 10.3 point update [production]
15:55 <moritzm> uploaded pypuppetdb 0.3.3-2~wmf+deb10u1 to apt.wikimedia.org [production]
15:55 <bblack> (log(n)) [production]
15:54 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2086:3318 T239453', diff saved to https://phabricator.wikimedia.org/P10414 and previous config saved to /var/cache/conftool/dbconfig/20200214-155443-marostegui.json [production]
15:52 <moritzm> uploaded pypuppetdb 0.3.3-2~wmf+deb9u1 to apt.wikimedia.org [production]
15:46 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Resync initialisesetting to try and pick up previoiusly deployed cirrus query routing changes (duration: 01m 05s) [production]