101-150 of 10000 results (18ms)
2020-04-13 §
19:06 <niedzielski> Morning SWAT done [production]
19:02 <niedzielski@deploy1001> Synchronized php-1.35.0-wmf.27/skins/MinervaNeue: SWAT: [[gerrit:588405|Update the icon glyph (T249864)]] (duration: 01m 00s) [production]
18:49 <niedzielski@deploy1001> Synchronized php-1.35.0-wmf.27/extensions/TwoColConflict: SWAT: [[gerrit:588370|Fix double HTML escaping of "copytext" lines in the diff (T249986)]] (duration: 01m 01s) [production]
17:01 <XioNoX> sample before any other border-in terms in eqiad [production]
16:57 <XioNoX> sample before any other border-in terms in esams [production]
16:50 <XioNoX> sample before any other border-in terms in dfw [production]
16:46 <XioNoX> sample before any other border-in terms in ulsfo [production]
16:36 <XioNoX> sample before any other border-in terms in eqsin [production]
16:36 <mholloway-shell@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . [production]
16:33 <mholloway-shell@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . [production]
16:31 <XioNoX> Sample all inbound v6 traffic on cr2-eqsin [production]
16:31 <cmjohnson1> replacing msw-c6-eqiad [production]
16:30 <mholloway-shell@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'wikifeeds' for release 'staging' . [production]
15:56 <marostegui> Deploy schema change on s4 codfw master - T250067 [production]
12:12 <vgutierrez> rolling upgrade to ats 8.0.7-rc0-1wm1 in eqsin and codfw [production]
11:58 <akosiaris@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'coredns' . [production]
11:57 <marostegui> Deploy schema change on eqiad s8 hosts - T250062 [production]
11:53 <akosiaris@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
11:53 <akosiaris@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
11:53 <akosiaris@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
11:53 <marostegui> Deploy schema change on codfw master (lag will appear on codfw) - T250062 [production]
11:15 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: efe2feb: robots.txt: Disable indexing user (sub)pages and draft-related pages on srwiki (T248860; take II) (duration: 00m 58s) [production]
11:14 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: efe2feb: robots.txt: Disable indexing user (sub)pages and draft-related pages on srwiki (T248860) (duration: 00m 58s) [production]
10:37 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:588383| Bumping portals to master (563985)]] (duration: 00m 58s) [production]
10:36 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:588383| Bumping portals to master (563985)]] (duration: 01m 00s) [production]
10:24 <mutante> depooled wdqs1004 by request because of high lag [production]
10:19 <marostegui> Kill updateSpecialPages.php --only=Fewestrevisions for s8 in mwmaint1002, the vslow host is lagging and creating errors [production]
10:12 <mutante> mwmaint1002 - sudo systemctl status mediawiki_job_translationnotifications-mediawikiwiki.service [production]
09:52 <Urbanecm> Rename user account Gerakiw@grwikimedia to Geraki@grwikimedia (T245911) [production]
09:47 <Urbanecm> mwscript createAndPromote.php --wiki=grwikimedia --force Gerakiw <redacted> (T245911) [production]
08:15 <marostegui> Remove grants for haproxy@10.64.37.15 from labsdb hosts T231280 [production]
07:50 <vgutierrez> enable memory tracking in ats-tls on cp1085 - T249335 [production]
07:43 <marostegui> Compress db1092 T232446 [production]
07:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Temporary pool db1111 in s8 API', diff saved to https://phabricator.wikimedia.org/P10964 and previous config saved to /var/cache/conftool/dbconfig/20200413-074158-marostegui.json [production]
07:40 <vgutierrez> rolling upgrade to ats 8.0.7-rc0-1wm1 in ulsfo [production]
07:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1092 T232446', diff saved to https://phabricator.wikimedia.org/P10963 and previous config saved to /var/cache/conftool/dbconfig/20200413-073939-marostegui.json [production]
07:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1110 T249973', diff saved to https://phabricator.wikimedia.org/P10962 and previous config saved to /var/cache/conftool/dbconfig/20200413-071740-marostegui.json [production]
06:51 <marostegui> Deploy schema changes on db1110 - T249973 [production]
06:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1110 T249973', diff saved to https://phabricator.wikimedia.org/P10961 and previous config saved to /var/cache/conftool/dbconfig/20200413-065022-marostegui.json [production]
06:36 <elukey> temporary stopped puppet on restbase2014 to avoid attempts to start cassandra on each run - T250050 [production]
06:23 <vgutierrez> upgrade to ats 8.0.7-rc0-1wm1 on cp[4026,4032,5006,5012] [production]
06:20 <vgutierrez> upload trafficserver 8.0.7-rc0-1wm1 to apt.wm.o (buster) [production]
05:25 <vgutierrez> restart varnish-fe on cp3050 [production]
2020-04-12 §
11:11 <vgutierrez> restart ats-tls on cp5008.eqsin.wmnet - T249335 [production]
10:18 <elukey> restart wdqs-updater on wdqs1004 (logs show no reports from the past hours, last one were stack traces related to a json decode failure) [production]
06:59 <dcausse> restarting blazegraph on wdqs1004 (T242453) [production]
06:35 <elukey@puppetmaster1001> conftool action : set/pooled=no; selector: name=restbase1025.eqiad.wmnet [production]
06:32 <elukey> powerdown restbase1025 - T250027 [production]
06:20 <elukey> powercycle restbase1025 (not reachable, serial console shows blank, racadm getsel reports errors with DIMM_B2) [production]
05:53 <bblack> pushing https://gerrit.wikimedia.org/r/588134 to cache_text [production]