1101-1150 of 10000 results (26ms)
2020-09-24 ยง
16:56 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
16:26 <jayme@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
16:26 <robh> properly pooled mw1360 this time T262151 [production]
16:18 <jayme@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'citoid' for release 'staging' . [production]
16:04 <XioNoX> pfw3-eqiad> restart security-log gracefully [production]
15:58 <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.10/extensions/AbuseFilter/includes/Hooks/AbuseFilterHookRunner.php: 5e88c36fa4111cde33dafb0d7ac31a854b95e5ea: HookRunner: onAbuseFilterGenerateUserVars should run generateUserVars (T263750) (duration: 01m 06s) [production]
15:46 <Urbanecm> Run `mwscript extensions/CentralAuth/maintenance/migrateAccount.php --wiki=simplewiki --username="Oversight~simplewiki"` (T263760) [production]
15:44 <Urbanecm> Run `mwscript extensions/CentralAuth/maintenance/migrateAccount.php --wiki=enwiki --username=Oversight` (T263760) [production]
15:43 <Urbanecm> Rename all local Oversight accounts but enwiki to Oversight~dbname, see task for full list (T263760) [production]
15:26 <marostegui@cumin1001> dbctl commit (dc=all): 'db2127 (re)pooling @ 100%: Slowly repool db2127 ', diff saved to https://phabricator.wikimedia.org/P12794 and previous config saved to /var/cache/conftool/dbconfig/20200924-152626-root.json [production]
15:15 <robh> mw1360 scap and repooled post work via T262151 [production]
15:11 <marostegui@cumin1001> dbctl commit (dc=all): 'db2127 (re)pooling @ 66%: Slowly repool db2127 ', diff saved to https://phabricator.wikimedia.org/P12793 and previous config saved to /var/cache/conftool/dbconfig/20200924-151120-root.json [production]
15:10 <jayme> switched zotero service-proxy listener to use TLS - T255869 [production]
15:00 <XioNoX> repool eqiad - T256112 [production]
14:56 <marostegui@cumin1001> dbctl commit (dc=all): 'db2127 (re)pooling @ 33%: Slowly repool db2127 ', diff saved to https://phabricator.wikimedia.org/P12792 and previous config saved to /var/cache/conftool/dbconfig/20200924-145617-root.json [production]
14:54 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:52 <robh@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:28 <XioNoX> [Netops] In window: turn VC-ports on/off for proper cabling: - T256112 [production]
14:19 <XioNoX> remove damping on anycast group for cr2-codfw [production]
14:18 <jayme> restart pybal on lvs1015.eqiad.wmnet,lvs2009.codfw.wmnet - T255869 [production]
14:16 <jayme> restart pybal on lvs1016.eqiad.wmnet,lvs2010.codfw.wmnet - T255869 [production]
14:16 <XioNoX> [Netops] Disable unused VC ports to not risk them going online at connect: - T256112 [production]
14:09 <jayme> running puppet on lvs servers - T255869 [production]
14:09 <cmjohnson1> removing the cable connected to FPC1:1/0 (DAC 3m) FPC8:1/0 (DAC 3m) [production]
13:58 <moritzm> upgrading mariadb on cloudcontrol-2001/2003/2004 [production]
13:52 <XioNoX> depool eqiad for row D recabling - T256112 [production]
13:32 <ottomata> Increased retention time for *.mediawiki.job.processMediaModeration topics in kafka main-eqiad and main-codfw to 31 days (as per request from Pchelolo ) [production]
13:22 <elukey> moved the hadoop cluster to puppet TLS certificates - T253957 [production]
13:17 <XioNoX> add damping to anycast BGP - T262372 [production]
12:58 <jayme> switched mathoid service-proxy listener to use TLS - T255875 [production]
12:50 <moritzm> upgrading bird on centtrallog1001 [production]
12:43 <gehel> restarting wdqs-categories on wdqs1009 [production]
12:43 <moritzm> installing netty-3.9 security updates [production]
12:42 <jgiannelos@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
12:30 <ema> upload@ulsfo: rolling varnish upgrade to 6.0.6-1wm1 T263557 [production]
12:29 <godog> swift codfw-prod: rebalance only, no weight change [production]
12:27 <kormat> powering off db2125 for maintenance T260670 [production]
12:25 <moritzm> installing xorg-server security updates [production]
12:09 <ema> text@ulsfo: rolling varnish upgrade to 6.0.6-1wm1 T263557 [production]
12:02 <ema> cp4022: upgrade varnish to 6.0.6-1wm1 T263557 [production]
11:40 <Urbanecm> EU B&C window done [production]
11:34 <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.10/extensions/Translate/tag/TPSection.php: fa4900e1e6022e645be12505de30b696a9769e77: Fix validation of translation unit section names (T263546) (duration: 01m 07s) [production]
11:25 <jbond42> re-enable puppet fleet wide [production]
11:22 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: fdab74c443bc3328856e8441f4d2df8bc57c6f54: Enable ContentTranslation in Bashkir, Urdu and Welsh WPs as a default tool (T258504; T260022; T260024) (duration: 01m 05s) [production]
11:21 <jbond42> disable puppet fleet wide to reduce log level on puppetdb [production]
11:14 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 90c72912f26d91df6d28b1efd64e366aaabc5357: Move DiscussionTools out of beta on arwiki, cswiki, huwiki (T249394); d8553f35b4dd581f67bd568d773ff65f316fbfd3: Simplify DiscussionTools config (duration: 01m 11s) [production]
11:06 <moritzm> installing imagemagick security updates on stretch [production]
11:02 <jbond42> re-enable puppet fleet wide [production]
10:51 <jbond42> disable puppet fleet wide to deploy a puppetmaster change [production]
10:49 <moritzm> installing libproxy security updates [production]