|
2020-02-05
§
|
| 10:10 |
<Urbanecm> |
Run mwscript deleteEqualMessages.php --delete to delete GrowthExperiments' message overrides (cswiki, viwiki, arwiki, kowiki) |
[production] |
| 09:57 |
<akosiaris> |
upload kubernetes 1.13.12 to apt.wikimedia.org stretch-wikimedia/main T244335 |
[production] |
| 09:51 |
<effie> |
install libmemcached-tools on mc-gp* servers - T240684 |
[production] |
| 09:05 |
<ema> |
add individual FortiGate IPs hitting ulsfo (currently cp4028) to vcl blocked_nets -- trying to identify problematic traffic T243634 |
[production] |
| 07:02 |
<marostegui> |
Replay s1 traffic on db1107 (10.4) T242702 |
[production] |
| 06:32 |
<elukey> |
force a puppet run on ores* hosts |
[production] |
| 06:12 |
<marostegui> |
Remove partitions from revision table db1098:3317 - T239453 |
[production] |
| 06:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1098:3317 - T239453', diff saved to https://phabricator.wikimedia.org/P10312 and previous config saved to /var/cache/conftool/dbconfig/20200205-060942-marostegui.json |
[production] |
| 06:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db2085:3311, db2086:3317 - T239453', diff saved to https://phabricator.wikimedia.org/P10311 and previous config saved to /var/cache/conftool/dbconfig/20200205-060911-marostegui.json |
[production] |
| 02:38 |
<cdanis> |
T243634 ✔️ cdanis@cp4030.ulsfo.wmnet ~ 🕤🍺 sudo varnish-frontend-restart |
[production] |
|
2020-02-04
§
|
| 22:35 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.18 refs T233866 |
[production] |
| 22:13 |
<twentyafterfour@deploy1001> |
Finished scap: testwikis wikis to 1.35.0-wmf.18 refs T233866 (duration: 32m 03s) |
[production] |
| 22:03 |
<cdanis@cumin2001> |
conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=eqiad |
[production] |
| 21:41 |
<twentyafterfour@deploy1001> |
Started scap: testwikis wikis to 1.35.0-wmf.18 refs T233866 |
[production] |
| 21:29 |
<twentyafterfour> |
preparing the new mediawiki branch for deployment to test wikis |
[production] |
| 20:31 |
<shdubsh> |
restart kartotherian on maps2001 |
[production] |
| 20:24 |
<shdubsh> |
temporarily enable access logs on maps2001 |
[production] |
| 20:20 |
<twentyafterfour> |
branching mediawiki to wmf/1.35.0-wmf.18 from commit 054dd94e97d6 - train blockers should be added as subtasks under T233866 |
[production] |
| 20:06 |
<marxarelli> |
temporarily holding 1.35.0-wmf.18 [T233866] branch cut and train due to concurrent maps prod issues |
[production] |
| 19:15 |
<mutante> |
cp3065 - powercycling |
[production] |
| 18:45 |
<cdanis@cumin2001> |
conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=eqiad |
[production] |
| 17:57 |
<cdanis> |
✔️ cdanis@mw1272.eqiad.wmnet ~ 🕐☕ sudo restart-php7.2-fpm |
[production] |
| 17:41 |
<akosiaris> |
reenable kartotherian on maps100* |
[production] |
| 17:34 |
<oblivian@cumin1001> |
conftool action : set/weight=15; selector: cluster=appserver,service=nginx,dc=eqiad,name=mw12[3-5].* |
[production] |
| 17:13 |
<_joe_> |
restarting php-fpm on mw126[1-3] |
[production] |
| 17:11 |
<_joe_> |
restarting php-fpm on mw1266-9 |
[production] |
| 17:10 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.16/includes/filerepo/file/ForeignDBFile.php: gerrit: 570089, ongoing incident (duration: 01m 04s) |
[production] |
| 17:07 |
<_joe_> |
restarted php-fpm on mw1265 witrh 80 workers (teh default) |
[production] |
| 17:07 |
<_joe_> |
restarted php-fpm on mw1264 witrh 240 workers |
[production] |
| 16:52 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.16/extensions/Wikibase: fix for the recent outage (duration: 01m 21s) |
[production] |
| 16:02 |
<ema> |
cp: rolling ats-backend-restart to unset Accept-Encoding before sending origin server requests T242478 |
[production] |
| 14:23 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 14:18 |
<akosiaris> |
deploy new wikifeeds chart that is consistent with the current scaffolding approach. No code deploy though. |
[production] |
| 14:17 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 14:16 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'wikifeeds' for release 'staging' . |
[production] |
| 14:07 |
<XioNoX> |
repool ulsfo |
[production] |
| 14:03 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) |
[production] |
| 14:00 |
<elukey@cumin1001> |
START - Cookbook sre.aqs.roll-restart |
[production] |
| 13:36 |
<XioNoX> |
restart cr3-ulsfo for software upgrade |
[production] |
| 13:23 |
<vgutierrez> |
upgrading acme-chief to version 0.22 - T240614 |
[production] |
| 13:10 |
<vgutierrez> |
uploaded acme-chief 0.22 to apt.wm.o (buster) - T240614 |
[production] |
| 13:09 |
<XioNoX> |
restart cr4-ulsfo for upgrade |
[production] |
| 12:49 |
<XioNoX> |
depool ulsfo for routers upgrade |
[production] |
| 10:35 |
<ema> |
cp4032: varnish-frontend-restart T243634 |
[production] |
| 09:08 |
<vgutierrez> |
manually refreshing OCSP stapling response for non-canonical-redirects-3 - T243948 |
[production] |
| 09:07 |
<marostegui> |
Upgrade s3 codfw master db2105 - T239791 |
[production] |
| 08:56 |
<marostegui> |
Deploy schema change on enwiki eqiad host by host - T243804 |
[production] |
| 08:46 |
<marostegui> |
Deploy schema change on enwiki codfw - T243804 |
[production] |
| 08:16 |
<marostegui> |
Deploy schema change on testwiki - T243804 |
[production] |
| 08:13 |
<marostegui> |
Deploy schema change on test2wiki - T243804 |
[production] |