|
2020-04-13
§
|
| 16:36 |
<mholloway-shell@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 16:33 |
<mholloway-shell@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 16:31 |
<XioNoX> |
Sample all inbound v6 traffic on cr2-eqsin |
[production] |
| 16:31 |
<cmjohnson1> |
replacing msw-c6-eqiad |
[production] |
| 16:30 |
<mholloway-shell@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'wikifeeds' for release 'staging' . |
[production] |
| 15:56 |
<marostegui> |
Deploy schema change on s4 codfw master - T250067 |
[production] |
| 12:12 |
<vgutierrez> |
rolling upgrade to ats 8.0.7-rc0-1wm1 in eqsin and codfw |
[production] |
| 11:58 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 11:57 |
<marostegui> |
Deploy schema change on eqiad s8 hosts - T250062 |
[production] |
| 11:53 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 11:53 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 11:53 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 11:53 |
<marostegui> |
Deploy schema change on codfw master (lag will appear on codfw) - T250062 |
[production] |
| 11:15 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: efe2feb: robots.txt: Disable indexing user (sub)pages and draft-related pages on srwiki (T248860; take II) (duration: 00m 58s) |
[production] |
| 11:14 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: efe2feb: robots.txt: Disable indexing user (sub)pages and draft-related pages on srwiki (T248860) (duration: 00m 58s) |
[production] |
| 10:37 |
<jdrewniak@deploy1001> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:588383| Bumping portals to master (563985)]] (duration: 00m 58s) |
[production] |
| 10:36 |
<jdrewniak@deploy1001> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:588383| Bumping portals to master (563985)]] (duration: 01m 00s) |
[production] |
| 10:24 |
<mutante> |
depooled wdqs1004 by request because of high lag |
[production] |
| 10:19 |
<marostegui> |
Kill updateSpecialPages.php --only=Fewestrevisions for s8 in mwmaint1002, the vslow host is lagging and creating errors |
[production] |
| 10:12 |
<mutante> |
mwmaint1002 - sudo systemctl status mediawiki_job_translationnotifications-mediawikiwiki.service |
[production] |
| 09:52 |
<Urbanecm> |
Rename user account Gerakiw@grwikimedia to Geraki@grwikimedia (T245911) |
[production] |
| 09:47 |
<Urbanecm> |
mwscript createAndPromote.php --wiki=grwikimedia --force Gerakiw <redacted> (T245911) |
[production] |
| 08:15 |
<marostegui> |
Remove grants for haproxy@10.64.37.15 from labsdb hosts T231280 |
[production] |
| 07:50 |
<vgutierrez> |
enable memory tracking in ats-tls on cp1085 - T249335 |
[production] |
| 07:43 |
<marostegui> |
Compress db1092 T232446 |
[production] |
| 07:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Temporary pool db1111 in s8 API', diff saved to https://phabricator.wikimedia.org/P10964 and previous config saved to /var/cache/conftool/dbconfig/20200413-074158-marostegui.json |
[production] |
| 07:40 |
<vgutierrez> |
rolling upgrade to ats 8.0.7-rc0-1wm1 in ulsfo |
[production] |
| 07:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1092 T232446', diff saved to https://phabricator.wikimedia.org/P10963 and previous config saved to /var/cache/conftool/dbconfig/20200413-073939-marostegui.json |
[production] |
| 07:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1110 T249973', diff saved to https://phabricator.wikimedia.org/P10962 and previous config saved to /var/cache/conftool/dbconfig/20200413-071740-marostegui.json |
[production] |
| 06:51 |
<marostegui> |
Deploy schema changes on db1110 - T249973 |
[production] |
| 06:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1110 T249973', diff saved to https://phabricator.wikimedia.org/P10961 and previous config saved to /var/cache/conftool/dbconfig/20200413-065022-marostegui.json |
[production] |
| 06:36 |
<elukey> |
temporary stopped puppet on restbase2014 to avoid attempts to start cassandra on each run - T250050 |
[production] |
| 06:23 |
<vgutierrez> |
upgrade to ats 8.0.7-rc0-1wm1 on cp[4026,4032,5006,5012] |
[production] |
| 06:20 |
<vgutierrez> |
upload trafficserver 8.0.7-rc0-1wm1 to apt.wm.o (buster) |
[production] |
| 05:25 |
<vgutierrez> |
restart varnish-fe on cp3050 |
[production] |
|
2020-04-12
§
|
| 11:11 |
<vgutierrez> |
restart ats-tls on cp5008.eqsin.wmnet - T249335 |
[production] |
| 10:18 |
<elukey> |
restart wdqs-updater on wdqs1004 (logs show no reports from the past hours, last one were stack traces related to a json decode failure) |
[production] |
| 06:59 |
<dcausse> |
restarting blazegraph on wdqs1004 (T242453) |
[production] |
| 06:35 |
<elukey@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=restbase1025.eqiad.wmnet |
[production] |
| 06:32 |
<elukey> |
powerdown restbase1025 - T250027 |
[production] |
| 06:20 |
<elukey> |
powercycle restbase1025 (not reachable, serial console shows blank, racadm getsel reports errors with DIMM_B2) |
[production] |
| 05:53 |
<bblack> |
pushing https://gerrit.wikimedia.org/r/588134 to cache_text |
[production] |
| 05:50 |
<vgutierrez> |
restart ats-tls on cp[1077,1081,1083,1085].eqiad.wmnet- T249335 |
[production] |
| 05:31 |
<bblack> |
pushing https://gerrit.wikimedia.org/r/588133 to cache_text |
[production] |
|
2020-04-11
§
|
| 19:52 |
<cdanis@cumin1001> |
dbctl commit (dc=all): 'slight deweight to db1111', diff saved to https://phabricator.wikimedia.org/P10960 and previous config saved to /var/cache/conftool/dbconfig/20200411-195235-cdanis.json |
[production] |
| 17:35 |
<cdanis@cumin1001> |
dbctl commit (dc=all): 's8: +weight db1111, -weight db1126', diff saved to https://phabricator.wikimedia.org/P10959 and previous config saved to /var/cache/conftool/dbconfig/20200411-173517-cdanis.json |
[production] |
| 15:39 |
<vgutierrez> |
restart ats-tls on cp[1077,1081,1083,1085].eqiad.wmnet- T249335 |
[production] |
| 09:30 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) |
[production] |
| 09:20 |
<elukey@cumin1001> |
START - Cookbook sre.presto.roll-restart-workers |
[production] |
| 07:01 |
<vgutierrez> |
restart ats-tls on cp[1079,1081,1083,1085].eqiad.wmnet- T249335 |
[production] |