6301-6350 of 10000 results (94ms)
2020-02-18 §
13:24 <XioNoX> reboot cr2-esams:re1 (backup) - T243080 [production]
13:23 <XioNoX> bump cost of eqiad-esams transport - T243080 [production]
13:10 <XioNoX> fail vrrp master to cr3-esams - T243080 [production]
12:58 <kartik@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . [production]
12:55 <Amir1> EU SWAT done [production]
12:53 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:572731|Add DiscussionTools to four wikis in hidden mode (T244870)]], take II (duration: 01m 03s) [production]
12:52 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:572731|Add DiscussionTools to four wikis in hidden mode (T244870)]] (duration: 01m 04s) [production]
12:45 <XioNoX> remove graceful-switchover and nonstop-routing from cr2-esams - T243080 [production]
12:36 <XioNoX> push new Junos to cr2-esams:re1 (backup RE, noop) - T243080 [production]
12:22 <ladsgroup@deploy1001> Synchronized wmf-config/Wikibase.php: SWAT: [[gerrit:569031|Wikibase: added config variables to configure entity sources (T242087)]], Part II (duration: 01m 03s) [production]
12:20 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:569031|Wikibase: added config variables to configure entity sources (T242087)]], Part I, take II (the cache issue) (duration: 01m 04s) [production]
12:18 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:569031|Wikibase: added config variables to configure entity sources (T242087)]], Part I (duration: 01m 06s) [production]
12:14 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:572628|Start reading for the new term store for clients up to Q1000 (T225057)]] (duration: 01m 05s) [production]
12:06 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 4b193dd: Increase Commons linkpurge rate limit for patrollers (T245214) (duration: 01m 31s) [production]
11:51 <kartik@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'cxserver' for release 'production' . [production]
11:48 <kartik@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . [production]
11:47 <kartik@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . [production]
11:43 <kartik@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . [production]
11:41 <kartik@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'cxserver' for release 'production' . [production]
11:35 <kartik@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . [production]
11:27 <jynus> reenabling prometheus exporter metadata user for prometheus1003 [production]
11:10 <jynus> temp. disabling prometheus exporter metadata user for prometheus1003 [production]
10:49 <marostegui@cumin1001> dbctl commit (dc=all): 'Increase API weight for db1107 15 -> 25 for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10445 and previous config saved to /var/cache/conftool/dbconfig/20200218-104958-marostegui.json [production]
09:27 <gehel> re-enable puppet on mw* - T222321 [production]
09:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1107 after temporary change optimizer options - T245489', diff saved to https://phabricator.wikimedia.org/P10444 and previous config saved to /var/cache/conftool/dbconfig/20200218-091343-marostegui.json [production]
09:09 <gehel> disabling puppet on mw* to deploy apache config change - T222321 [production]
09:07 <volans> rm /var/log/exim4/paniclog on cumin1001 to clear OOM from last week error [production]
08:59 <marostegui> Remove wikiadmin2 grants from es1 T243512 [production]
08:59 <marostegui> Remove wikiadmin2 grants from es1 [production]
08:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1107 after temporary change optimizer options', diff saved to https://phabricator.wikimedia.org/P10443 and previous config saved to /var/cache/conftool/dbconfig/20200218-085713-marostegui.json [production]
08:23 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1107 after temporary change optimizer options - T245489', diff saved to https://phabricator.wikimedia.org/P10442 and previous config saved to /var/cache/conftool/dbconfig/20200218-082306-marostegui.json [production]
08:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1107 after temporary change optimizer options - T245489', diff saved to https://phabricator.wikimedia.org/P10441 and previous config saved to /var/cache/conftool/dbconfig/20200218-080952-marostegui.json [production]
08:08 <marostegui> Restart MySQL to pick up optimizer_switch changes - T245489 [production]
08:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1107 to temporary change optimizer options - T245489', diff saved to https://phabricator.wikimedia.org/P10440 and previous config saved to /var/cache/conftool/dbconfig/20200218-080623-marostegui.json [production]
07:34 <elukey> powercycle analytics1065 (crashed hours ago, no mgmt console available, no ssh) [production]
06:39 <marostegui> Remove wikiadmin2 from pc1007, pc1008, pc1009 and pc1010 T243512 [production]
06:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Increase weight for db1107 100 -> 200 for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10439 and previous config saved to /var/cache/conftool/dbconfig/20200218-063819-marostegui.json [production]
06:27 <marostegui> Stop haproxy on dbproxy1007 - T245385 [production]
06:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1107 with weight 100 and weight 10 in API for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10438 and previous config saved to /var/cache/conftool/dbconfig/20200218-062459-marostegui.json [production]
06:09 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
06:08 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
2020-02-17 §
19:56 <cdanis> finish enabling TCP-MSS clamping in eqiad [production]
19:49 <cdanis> s/no-op// [production]
19:49 <cdanis> no-op enable TCP-MSS clamping on eqord and eqiad [production]
19:33 <cdanis> no-op enable flowspec change on cr2-eqord and cr2-eqiad [production]
18:25 <elukey> restart kafka on kafka-jumbo1001 to pick up new openjdk updates [production]
17:25 <bblack> GRE MTU mitigations applied to esams cp hosts only - T232602 [production]
15:55 <ayounsi@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
15:50 <ayounsi@cumin1001> START - Cookbook sre.ganeti.makevm [production]
15:48 <ayounsi@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]