3201-3250 of 10000 results (81ms)
2020-02-05 §
09:05 <ema> add individual FortiGate IPs hitting ulsfo (currently cp4028) to vcl blocked_nets -- trying to identify problematic traffic T243634 [production]
07:02 <marostegui> Replay s1 traffic on db1107 (10.4) T242702 [production]
06:32 <elukey> force a puppet run on ores* hosts [production]
06:12 <marostegui> Remove partitions from revision table db1098:3317 - T239453 [production]
06:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1098:3317 - T239453', diff saved to https://phabricator.wikimedia.org/P10312 and previous config saved to /var/cache/conftool/dbconfig/20200205-060942-marostegui.json [production]
06:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2085:3311, db2086:3317 - T239453', diff saved to https://phabricator.wikimedia.org/P10311 and previous config saved to /var/cache/conftool/dbconfig/20200205-060911-marostegui.json [production]
02:38 <cdanis> T243634 ✔️ cdanis@cp4030.ulsfo.wmnet ~ 🕤🍺 sudo varnish-frontend-restart [production]
2020-02-04 §
22:35 <twentyafterfour@deploy1001> rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.18 refs T233866 [production]
22:13 <twentyafterfour@deploy1001> Finished scap: testwikis wikis to 1.35.0-wmf.18 refs T233866 (duration: 32m 03s) [production]
22:03 <cdanis@cumin2001> conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=eqiad [production]
21:41 <twentyafterfour@deploy1001> Started scap: testwikis wikis to 1.35.0-wmf.18 refs T233866 [production]
21:29 <twentyafterfour> preparing the new mediawiki branch for deployment to test wikis [production]
20:31 <shdubsh> restart kartotherian on maps2001 [production]
20:24 <shdubsh> temporarily enable access logs on maps2001 [production]
20:20 <twentyafterfour> branching mediawiki to wmf/1.35.0-wmf.18 from commit 054dd94e97d6 - train blockers should be added as subtasks under T233866 [production]
20:06 <marxarelli> temporarily holding 1.35.0-wmf.18 [T233866] branch cut and train due to concurrent maps prod issues [production]
19:15 <mutante> cp3065 - powercycling [production]
18:45 <cdanis@cumin2001> conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=eqiad [production]
17:57 <cdanis> ✔️ cdanis@mw1272.eqiad.wmnet ~ 🕐☕ sudo restart-php7.2-fpm [production]
17:41 <akosiaris> reenable kartotherian on maps100* [production]
17:34 <oblivian@cumin1001> conftool action : set/weight=15; selector: cluster=appserver,service=nginx,dc=eqiad,name=mw12[3-5].* [production]
17:13 <_joe_> restarting php-fpm on mw126[1-3] [production]
17:11 <_joe_> restarting php-fpm on mw1266-9 [production]
17:10 <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.16/includes/filerepo/file/ForeignDBFile.php: gerrit: 570089, ongoing incident (duration: 01m 04s) [production]
17:07 <_joe_> restarted php-fpm on mw1265 witrh 80 workers (teh default) [production]
17:07 <_joe_> restarted php-fpm on mw1264 witrh 240 workers [production]
16:52 <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.16/extensions/Wikibase: fix for the recent outage (duration: 01m 21s) [production]
16:02 <ema> cp: rolling ats-backend-restart to unset Accept-Encoding before sending origin server requests T242478 [production]
14:23 <akosiaris@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . [production]
14:18 <akosiaris> deploy new wikifeeds chart that is consistent with the current scaffolding approach. No code deploy though. [production]
14:17 <akosiaris@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . [production]
14:16 <akosiaris@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'wikifeeds' for release 'staging' . [production]
14:07 <XioNoX> repool ulsfo [production]
14:03 <elukey@cumin1001> END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) [production]
14:00 <elukey@cumin1001> START - Cookbook sre.aqs.roll-restart [production]
13:36 <XioNoX> restart cr3-ulsfo for software upgrade [production]
13:23 <vgutierrez> upgrading acme-chief to version 0.22 - T240614 [production]
13:10 <vgutierrez> uploaded acme-chief 0.22 to apt.wm.o (buster) - T240614 [production]
13:09 <XioNoX> restart cr4-ulsfo for upgrade [production]
12:49 <XioNoX> depool ulsfo for routers upgrade [production]
10:35 <ema> cp4032: varnish-frontend-restart T243634 [production]
09:08 <vgutierrez> manually refreshing OCSP stapling response for non-canonical-redirects-3 - T243948 [production]
09:07 <marostegui> Upgrade s3 codfw master db2105 - T239791 [production]
08:56 <marostegui> Deploy schema change on enwiki eqiad host by host - T243804 [production]
08:46 <marostegui> Deploy schema change on enwiki codfw - T243804 [production]
08:16 <marostegui> Deploy schema change on testwiki - T243804 [production]
08:13 <marostegui> Deploy schema change on test2wiki - T243804 [production]
07:36 <marostegui> Upgrade Mariadb on db1107 from 10.4.11 to 10.4.12 T242702 [production]
07:15 <marostegui> Compress db1126 - T232446 [production]
07:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1126 - T232446', diff saved to https://phabricator.wikimedia.org/P10302 and previous config saved to /var/cache/conftool/dbconfig/20200204-071420-marostegui.json [production]