1551-1600 of 10000 results (59ms)
2020-04-13 §
10:37 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:588383| Bumping portals to master (563985)]] (duration: 00m 58s) [production]
10:36 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:588383| Bumping portals to master (563985)]] (duration: 01m 00s) [production]
10:24 <mutante> depooled wdqs1004 by request because of high lag [production]
10:19 <marostegui> Kill updateSpecialPages.php --only=Fewestrevisions for s8 in mwmaint1002, the vslow host is lagging and creating errors [production]
10:12 <mutante> mwmaint1002 - sudo systemctl status mediawiki_job_translationnotifications-mediawikiwiki.service [production]
09:52 <Urbanecm> Rename user account Gerakiw@grwikimedia to Geraki@grwikimedia (T245911) [production]
09:47 <Urbanecm> mwscript createAndPromote.php --wiki=grwikimedia --force Gerakiw <redacted> (T245911) [production]
08:15 <marostegui> Remove grants for haproxy@10.64.37.15 from labsdb hosts T231280 [production]
07:50 <vgutierrez> enable memory tracking in ats-tls on cp1085 - T249335 [production]
07:43 <marostegui> Compress db1092 T232446 [production]
07:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Temporary pool db1111 in s8 API', diff saved to https://phabricator.wikimedia.org/P10964 and previous config saved to /var/cache/conftool/dbconfig/20200413-074158-marostegui.json [production]
07:40 <vgutierrez> rolling upgrade to ats 8.0.7-rc0-1wm1 in ulsfo [production]
07:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1092 T232446', diff saved to https://phabricator.wikimedia.org/P10963 and previous config saved to /var/cache/conftool/dbconfig/20200413-073939-marostegui.json [production]
07:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1110 T249973', diff saved to https://phabricator.wikimedia.org/P10962 and previous config saved to /var/cache/conftool/dbconfig/20200413-071740-marostegui.json [production]
06:51 <marostegui> Deploy schema changes on db1110 - T249973 [production]
06:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1110 T249973', diff saved to https://phabricator.wikimedia.org/P10961 and previous config saved to /var/cache/conftool/dbconfig/20200413-065022-marostegui.json [production]
06:36 <elukey> temporary stopped puppet on restbase2014 to avoid attempts to start cassandra on each run - T250050 [production]
06:23 <vgutierrez> upgrade to ats 8.0.7-rc0-1wm1 on cp[4026,4032,5006,5012] [production]
06:20 <vgutierrez> upload trafficserver 8.0.7-rc0-1wm1 to apt.wm.o (buster) [production]
05:25 <vgutierrez> restart varnish-fe on cp3050 [production]
2020-04-12 §
11:11 <vgutierrez> restart ats-tls on cp5008.eqsin.wmnet - T249335 [production]
10:18 <elukey> restart wdqs-updater on wdqs1004 (logs show no reports from the past hours, last one were stack traces related to a json decode failure) [production]
06:59 <dcausse> restarting blazegraph on wdqs1004 (T242453) [production]
06:35 <elukey@puppetmaster1001> conftool action : set/pooled=no; selector: name=restbase1025.eqiad.wmnet [production]
06:32 <elukey> powerdown restbase1025 - T250027 [production]
06:20 <elukey> powercycle restbase1025 (not reachable, serial console shows blank, racadm getsel reports errors with DIMM_B2) [production]
05:53 <bblack> pushing https://gerrit.wikimedia.org/r/588134 to cache_text [production]
05:50 <vgutierrez> restart ats-tls on cp[1077,1081,1083,1085].eqiad.wmnet- T249335 [production]
05:31 <bblack> pushing https://gerrit.wikimedia.org/r/588133 to cache_text [production]
2020-04-11 §
19:52 <cdanis@cumin1001> dbctl commit (dc=all): 'slight deweight to db1111', diff saved to https://phabricator.wikimedia.org/P10960 and previous config saved to /var/cache/conftool/dbconfig/20200411-195235-cdanis.json [production]
17:35 <cdanis@cumin1001> dbctl commit (dc=all): 's8: +weight db1111, -weight db1126', diff saved to https://phabricator.wikimedia.org/P10959 and previous config saved to /var/cache/conftool/dbconfig/20200411-173517-cdanis.json [production]
15:39 <vgutierrez> restart ats-tls on cp[1077,1081,1083,1085].eqiad.wmnet- T249335 [production]
09:30 <elukey@cumin1001> END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) [production]
09:20 <elukey@cumin1001> START - Cookbook sre.presto.roll-restart-workers [production]
07:01 <vgutierrez> restart ats-tls on cp[1079,1081,1083,1085].eqiad.wmnet- T249335 [production]
2020-04-10 §
21:12 <cdanis@cumin1001> dbctl commit (dc=all): 'db1111 seems overloaded', diff saved to https://phabricator.wikimedia.org/P10954 and previous config saved to /var/cache/conftool/dbconfig/20200410-211202-cdanis.json [production]
19:37 <cdanis> cdanis@re0.cr1-codfw> clear bfd session address 208.80.153.220 [production]
15:03 <vgutierrez> restart ats-tls on cp1083 and cp1085 - T249335 [production]
13:14 <hashar@deploy1001> Finished deploy [zuul/deploy@4a69913]: (no justification provided) (duration: 00m 40s) [production]
13:14 <hashar@deploy1001> Started deploy [zuul/deploy@4a69913]: (no justification provided) [production]
13:12 <mutante> restarted and re-armed keyholder on deploy1001 to pick up changes for zuul scap deploy [production]
12:12 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
12:11 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
12:10 <mutante> Creating VM people1002.eqiad.wmnet in cluster ganeti01.svc.eqiad.wmnet with row=A vcpus=1 memory=2GB disk=80GB link=private. (T249907) [production]
12:10 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
12:10 <mutante> Creating VM people1002.eqiad.wmnet in cluster ganeti01.svc.eqiad.wmnet with row=A vcpus=1 memory=2GB disk=80GB link=private. This may take a few minutes. [production]
12:10 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
12:09 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
12:09 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
11:47 <akosiaris@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'mathoid' for release 'canary' . [production]