1151-1200 of 10000 results (25ms)
2020-09-24 ยง
18:58 <tchanders@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:627481|Enable Special:Investigate on itwiki and svwiki (T262436)]] (duration: 01m 05s) [production]
18:01 <mutante> temp. disabled puppet on install4001/install5001 - applying install_server role to new servers, starting with install3001 [production]
17:57 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
17:57 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
17:57 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
17:57 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
17:57 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
17:57 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
17:24 <mholloway-shell@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
17:21 <jbond42> enable puppet fleet wide post update puppetdb postgres logging [production]
17:19 <mholloway-shell@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
17:17 <jayme@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
17:16 <jayme@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
17:15 <mholloway-shell@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
17:15 <jbond42> disable puppet fleet wide to update puppetdb postgres loggin [production]
17:14 <hnowlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
17:14 <hnowlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'changeprop' for release 'production' . [production]
17:14 <mholloway-shell@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
17:11 <mholloway-shell@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
17:11 <hnowlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
17:11 <hnowlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
17:09 <mholloway-shell@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
17:04 <mutante> syncing facts to puppet compiler hosts [production]
17:01 <jayme@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
17:00 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:56 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
16:26 <jayme@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
16:26 <robh> properly pooled mw1360 this time T262151 [production]
16:18 <jayme@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'citoid' for release 'staging' . [production]
16:04 <XioNoX> pfw3-eqiad> restart security-log gracefully [production]
15:58 <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.10/extensions/AbuseFilter/includes/Hooks/AbuseFilterHookRunner.php: 5e88c36fa4111cde33dafb0d7ac31a854b95e5ea: HookRunner: onAbuseFilterGenerateUserVars should run generateUserVars (T263750) (duration: 01m 06s) [production]
15:46 <Urbanecm> Run `mwscript extensions/CentralAuth/maintenance/migrateAccount.php --wiki=simplewiki --username="Oversight~simplewiki"` (T263760) [production]
15:44 <Urbanecm> Run `mwscript extensions/CentralAuth/maintenance/migrateAccount.php --wiki=enwiki --username=Oversight` (T263760) [production]
15:43 <Urbanecm> Rename all local Oversight accounts but enwiki to Oversight~dbname, see task for full list (T263760) [production]
15:26 <marostegui@cumin1001> dbctl commit (dc=all): 'db2127 (re)pooling @ 100%: Slowly repool db2127 ', diff saved to https://phabricator.wikimedia.org/P12794 and previous config saved to /var/cache/conftool/dbconfig/20200924-152626-root.json [production]
15:15 <robh> mw1360 scap and repooled post work via T262151 [production]
15:11 <marostegui@cumin1001> dbctl commit (dc=all): 'db2127 (re)pooling @ 66%: Slowly repool db2127 ', diff saved to https://phabricator.wikimedia.org/P12793 and previous config saved to /var/cache/conftool/dbconfig/20200924-151120-root.json [production]
15:10 <jayme> switched zotero service-proxy listener to use TLS - T255869 [production]
15:00 <XioNoX> repool eqiad - T256112 [production]
14:56 <marostegui@cumin1001> dbctl commit (dc=all): 'db2127 (re)pooling @ 33%: Slowly repool db2127 ', diff saved to https://phabricator.wikimedia.org/P12792 and previous config saved to /var/cache/conftool/dbconfig/20200924-145617-root.json [production]
14:54 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:52 <robh@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:28 <XioNoX> [Netops] In window: turn VC-ports on/off for proper cabling: - T256112 [production]
14:19 <XioNoX> remove damping on anycast group for cr2-codfw [production]
14:18 <jayme> restart pybal on lvs1015.eqiad.wmnet,lvs2009.codfw.wmnet - T255869 [production]
14:16 <jayme> restart pybal on lvs1016.eqiad.wmnet,lvs2010.codfw.wmnet - T255869 [production]
14:16 <XioNoX> [Netops] Disable unused VC ports to not risk them going online at connect: - T256112 [production]
14:09 <jayme> running puppet on lvs servers - T255869 [production]
14:09 <cmjohnson1> removing the cable connected to FPC1:1/0 (DAC 3m) FPC8:1/0 (DAC 3m) [production]
13:58 <moritzm> upgrading mariadb on cloudcontrol-2001/2003/2004 [production]