5501-5550 of 10000 results (66ms)
2022-01-20 §
07:01 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance [production]
07:01 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
07:01 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
07:00 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance [production]
07:00 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance [production]
07:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T285149)', diff saved to https://phabricator.wikimedia.org/P18902 and previous config saved to /var/cache/conftool/dbconfig/20220120-070052-marostegui.json [production]
06:55 <marostegui@cumin1001> dbctl commit (dc=all): 'db1180 (re)pooling @ 1%: repooling after reimage', diff saved to https://phabricator.wikimedia.org/P18901 and previous config saved to /var/cache/conftool/dbconfig/20220120-065551-root.json [production]
06:47 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1180.eqiad.wmnet with OS bullseye [production]
06:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P18900 and previous config saved to /var/cache/conftool/dbconfig/20220120-064547-marostegui.json [production]
06:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P18899 and previous config saved to /var/cache/conftool/dbconfig/20220120-063042-marostegui.json [production]
06:17 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db1180.eqiad.wmnet with OS bullseye [production]
06:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T285149)', diff saved to https://phabricator.wikimedia.org/P18898 and previous config saved to /var/cache/conftool/dbconfig/20220120-061538-marostegui.json [production]
06:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1180 T299479', diff saved to https://phabricator.wikimedia.org/P18897 and previous config saved to /var/cache/conftool/dbconfig/20220120-061529-marostegui.json [production]
06:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1098:3317 (T285149)', diff saved to https://phabricator.wikimedia.org/P18896 and previous config saved to /var/cache/conftool/dbconfig/20220120-061407-marostegui.json [production]
06:14 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance [production]
06:14 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance [production]
2022-01-19 §
23:35 <mutante> deploy1002 - checked freshly generated cert in /etc/helmfile-defaults/private/main_services/miscweb/eqiad.yaml with 'openssl x509 -noout -text -in .. | grep DNS'. now has static-bz on it. (T281538) [production]
23:35 <mutante> puppetmaster1001 - revoked puppet cert miscweb.discovery.wmnet; updated kube_services.crts.yaml to include static-bugzilla.wikimedia.org, removed miscweb.discovery.wmnet.crt and .csr.pem, used cergen to check and regenerate cert, committed in private repo, ran puppet on deploy1001 - checked cert in /etc/helmfile-defaults/private/main_services/miscweb/eqiad.yaml with 'openssl x509 [production]
21:43 <aqu@deploy1002> Finished deploy [airflow-dags/analytics-test@3ad07a0]: (no justification provided) (duration: 00m 26s) [production]
21:42 <aqu@deploy1002> Started deploy [airflow-dags/analytics-test@3ad07a0]: (no justification provided) [production]
20:52 <Krinkle> depool mw1340 (api_appserver) for performance and php-apcu testing [production]
20:13 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
20:12 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
20:12 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
20:11 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
20:08 <jhuneidi@deploy1002> Synchronized php: group1 wikis to 1.38.0-wmf.18 refs T293959 (duration: 00m 49s) [production]
20:08 <jhuneidi@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.38.0-wmf.18 refs T293959 [production]
20:04 <jhathaway> rebooting mx1001 to debug conntrack [production]
19:52 <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.18/tests/phpunit/structure/SettingsTest.php: ed5e634772d2821c6f61903f7341eef4f2fc4337: First pass on creating config-schema.yaml (duration: 00m 49s) [production]
19:49 <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.18/includes/: ed5e634772d2821c6f61903f7341eef4f2fc4337: First pass on creating config-schema.yaml (duration: 01m 02s) [production]
19:47 <herron@puppetmaster1001> conftool action : set/pooled=no; selector: name=logstash1009.eqiad.wmnet [production]
19:47 <herron@puppetmaster1001> conftool action : set/pooled=no; selector: name=logstash1008.eqiad.wmnet [production]
19:47 <herron@puppetmaster1001> conftool action : set/pooled=no; selector: name=logstash1007.eqiad.wmnet [production]
19:45 <herron@puppetmaster1001> conftool action : set/pooled=no; selector: name=logstash2006.codfw.wmnet [production]
19:45 <herron@puppetmaster1001> conftool action : set/pooled=no; selector: name=logstash2005.codfw.wmnet [production]
19:45 <herron@puppetmaster1001> conftool action : set/pooled=no; selector: name=logstash2004.codfw.wmnet [production]
19:35 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
19:34 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
19:34 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
19:33 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
19:32 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: name=restbase2016.codfw.wmnet [production]
19:31 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host restbase2016.codfw.wmnet with OS buster [production]
19:17 <cjming@deploy1002> Synchronized wmf-config/config: Config: [[gerrit:755038|Update config for pilot wikis: (T298519)]] (duration: 00m 49s) [production]
19:13 <cjming@deploy1002> Synchronized wmf-config/config: message (duration: 00m 50s) [production]
19:13 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
19:12 <cjming@deploy1002> Synchronized wmf-config/config/foundationwiki.yaml: Config: [[gerrit:755038|Update config for pilot wikis: (T298519)]] (duration: 00m 49s) [production]
19:11 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
19:11 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
19:11 <cjming@deploy1002> Synchronized wmf-config/config/viwiki.yaml: Config: [[gerrit:755038|Update config for pilot wikis: (T298519)]] (duration: 00m 49s) [production]
19:10 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]