4001-4050 of 10000 results (42ms)
2021-09-13 §
15:13 <jelto@cumin2002> conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=restbase-async [production]
15:13 <legoktm> (cotd.) box-constraints|similar-users|termbox|thanos-query|thanos-swift|wdqs|wdqs-internal|wikifeeds|zotero) [production]
15:13 <rzl> (contd.) box-constraints|similar-users|termbox|thanos-query|thanos-swift|wdqs|wdqs-internal|wikifeeds|zotero) [production]
15:12 <jelto@cumin2002> conftool action : set/pooled=true; selector: name=codfw,dnsdisc=(apertium|api-gateway|citoid|cxserver|echostore|eventgate-analytics|eventgate-analytics-external|eventgate-logging-external|eventgate-main|eventstreams|eventstreams-internal|kartotherian|linkrecommendation|mathoid|mobileapps|ores|proton|push-notifications|recommendation-api|restbase|restbase-async|schema|search|sessionstore|shellbox|shell [production]
15:02 <jelto@cumin2002> END (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0) [production]
15:02 <topranks> Restarting unused line-card FPC 1 in cr2-codfw in attempt to clear alarm. [production]
14:56 <jelto@cumin2002> START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep [production]
14:44 <herron> drained mx2001 mail queue to mx1001 T286911 [production]
14:38 <dcausse> restarting wdqs-updater.service on all wdqs servers [production]
14:21 <jelto@cumin2002> END (PASS) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=0) [production]
14:20 <jelto@cumin2002> START - Cookbook sre.switchdc.services.02-restore-ttl [production]
14:13 <jelto@cumin2002> END (PASS) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=0) [production]
14:13 <legoktm> (cotd.) ternal, eventgate-main, wikifeeds, eventstreams-internal, eventgate-analytics-external: codfw => eqiad [production]
14:12 <jelto@cumin2002> Switching services echostore, termbox, cxserver, eventstreams, search, ores, mathoid, schema, push-notifications, thanos-swift, wdqs, sessionstore, restbase, wdqs-internal, apertium, eventgate-analytics, citoid, api-gateway, restbase-async, proton, linkrecommendation, thanos-query, shellbox, kartotherian, mobileapps, recommendation-api, zotero, similar-users, shellbox-constraints, eventgate-logging-ex [production]
14:12 <jelto@cumin2002> START - Cookbook sre.switchdc.services.01-switch-dc [production]
14:11 <jelto@cumin2002> END (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0) [production]
14:05 <jelto@cumin2002> START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep [production]
14:03 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host durum3002.esams.wmnet [production]
13:51 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm for new host durum3002.esams.wmnet [production]
13:50 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host durum3001.esams.wmnet [production]
13:39 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm for new host durum3001.esams.wmnet [production]
13:36 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host durum2002.codfw.wmnet [production]
13:21 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm for new host durum2002.codfw.wmnet [production]
13:20 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host durum2001.codfw.wmnet [production]
13:08 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm for new host durum2001.codfw.wmnet [production]
12:09 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
12:03 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
11:32 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:27 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:26 <kostajh> European mid-day backport window deploys done [production]
11:24 <kharlan@deploy1002> Synchronized wmf-config: Config: [[gerrit:713553|WikimediaEvents: Remove UnderstandingFirstDay config]] (duration: 00m 59s) [production]
10:51 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2002.codfw.wmnet [production]
10:43 <jmm@cumin2002> START - Cookbook sre.hosts.decommission for hosts testvm2002.codfw.wmnet [production]
10:15 <volans@cumin1001> END (FAIL) - Cookbook sre.experimental.reimage (exit_code=93) for host mw1414.eqiad.wmnet [production]
09:33 <volans> restarting tcpircbot-logmsgbot on alert1001, not relying messages [production]
09:18 <elukey> upgrade rsyslog* on ml-serve* nodes to 8.1901.0-1+wmf2 [production]
09:16 <godog> swift eqiad-prod: add weight to ms-be10[64-67] - T290546 [production]
09:11 <moritzm> reimaging sretest1002 [production]
09:11 <elukey> upload rsyslog* 8.1901.0-1+wmf2 to buster-wikimedia component/rsyslog-k8s - T277739 [production]
08:16 <godog> bump +100G prometheus/ops codfw [production]
2021-09-12 §
18:33 <vgutierrez> restart varnish-fe on cp3061, cp3063 and cp3065 [production]
18:29 <vgutierrez> restart varnish on cp3055 [production]
18:26 <vgutierrez> restart varnish on cp3057 [production]
04:53 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
04:52 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
2021-09-11 §
19:02 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 27814b8eaacb5ba2fee1b6167a36ea14356a1ecf: testwiki: Fully remove securepoll-related groups (T290808) (duration: 00m 57s) [production]
18:34 <urbanecm> [urbanecm@mwmaint2002 ~]$ mwscript emptyUserGroup.php --wiki=testwiki {electionadmin,electcomm} # T290808 [production]
18:31 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 908bbf35235ea4129795dfbf4c0e646440152e18: Revert "test: Add electcomm and electionadmin groups" (T290808) (duration: 00m 58s) [production]
2021-09-10 §
21:28 <legoktm@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'shellbox-syntaxhighlight' for release 'main' . [production]
21:27 <legoktm@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'shellbox-syntaxhighlight' for release 'main' . [production]