4901-4950 of 10000 results (72ms)
2019-11-21 ยง
16:48 <sbassett@deploy1001> Finished scap: Deploying T238451 (ext:AbuseFilter), running scap sync for i18n issues. (duration: 16m 42s) [production]
16:31 <sbassett@deploy1001> Started scap: Deploying T238451 (ext:AbuseFilter), running scap sync for i18n issues. [production]
15:54 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . [production]
15:42 <mforns@deploy1001> Finished deploy [analytics/refinery@7f32472]: deploying analytics refinery (after refinery-source v0.0.107) (duration: 10m 50s) [production]
15:31 <mforns@deploy1001> Started deploy [analytics/refinery@7f32472]: deploying analytics refinery (after refinery-source v0.0.107) [production]
15:30 <ema> pool cp1079 with ATS backend T227432 [production]
15:22 <mholloway-shell@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . [production]
15:19 <mholloway-shell@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . [production]
15:13 <akosiaris> purge https://releases.wikimedia.org/charts/eventgate-0.0.13.tgz, https://releases.wikimedia.org/charts/ and https://releases.wikimedia.org/charts/index.yaml [production]
15:09 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:07 <bblack> DONE testing deployment software changes on authdns cluster, back to normal [production]
15:07 <ema@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:49 <ema> depool cp1079 and reimage as text_ats T227432 [production]
14:47 <onimisionipe@deploy1001> Finished deploy [wdqs/wdqs@db43901]: Agent filter changes (duration: 18m 33s) [production]
14:43 <bblack> testing deployment software changes on authdns cluster, please hold dns changes for a few! [production]
14:41 <thcipriani> restarting Jenkins for update [production]
14:28 <onimisionipe@deploy1001> Started deploy [wdqs/wdqs@db43901]: Agent filter changes [production]
13:59 <ema> pool cp1077 with ATS backend T227432 [production]
13:41 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:39 <ema@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:20 <ema> depool cp1077 and reimage as text_ats T227432 [production]
11:53 <reedy@deploy1001> Finished scap: T234450 (duration: 19m 20s) [production]
11:42 <effie> enable puppet on all mw hosts [production]
11:33 <reedy@deploy1001> Started scap: T234450 [production]
11:09 <Urbanecm> EU SWAT done [production]
11:08 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: e4861ec: Set correct language for shywiktionary (T238105) (duration: 00m 52s) [production]
11:06 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 68d2003: Restrict editing CNBanner namespace to autoconfirmed on metawiki (T238723) (duration: 00m 54s) [production]
11:05 <effie> disable puppet on mw[1-2]* [production]
10:49 <volans> restarting tcpircbot-logmsgbot on icinga1001, has failed to log some messages, no useful log on the host [production]
10:22 <ema> pool cp2023 with Varnish backend T238817 T227432 [production]
10:17 <arturo> update buster-wikimedia thirdparty/kubeadm-k8s packages (newer version will be used to handle T238654) [production]
09:54 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1090:331{2,7} after upgrade', diff saved to https://phabricator.wikimedia.org/P9714 and previous config saved to /var/cache/conftool/dbconfig/20191121-095401-marostegui.json [production]
09:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1090:331{2,7} after upgrade', diff saved to https://phabricator.wikimedia.org/P9713 and previous config saved to /var/cache/conftool/dbconfig/20191121-093958-marostegui.json [production]
09:39 <ema> depool cp2023 and reimage back as varnish-be T238817 T227432 [production]
09:38 <marostegui> Stop MySQL on db1067 - T238297 [production]
09:27 <marostegui> Upgrade db1090:3312, db1090:3317 [production]
09:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1090:3312, db1090:3317 for upgrade', diff saved to https://phabricator.wikimedia.org/P9712 and previous config saved to /var/cache/conftool/dbconfig/20191121-092554-marostegui.json [production]
09:08 <akosiaris@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'wikifeeds' for release 'staging' . [production]
09:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1079 after upgrade', diff saved to https://phabricator.wikimedia.org/P9711 and previous config saved to /var/cache/conftool/dbconfig/20191121-090623-marostegui.json [production]
09:03 <akosiaris@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'wikifeeds' for release 'staging' . [production]
08:58 <akosiaris@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' . [production]
08:56 <marostegui@cumin1001> dbctl commit (dc=all): 'More traffic to db1079 after upgrade', diff saved to https://phabricator.wikimedia.org/P9710 and previous config saved to /var/cache/conftool/dbconfig/20191121-085644-marostegui.json [production]
08:53 <akosiaris@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' . [production]
08:45 <marostegui@cumin1001> dbctl commit (dc=all): 'More traffic to db1079 after upgrade', diff saved to https://phabricator.wikimedia.org/P9709 and previous config saved to /var/cache/conftool/dbconfig/20191121-084500-marostegui.json [production]
08:33 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1079 after upgrade', diff saved to https://phabricator.wikimedia.org/P9708 and previous config saved to /var/cache/conftool/dbconfig/20191121-083322-marostegui.json [production]
08:21 <marostegui> Upgrade db1079 [production]
08:21 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1079 for upgrade', diff saved to https://phabricator.wikimedia.org/P9707 and previous config saved to /var/cache/conftool/dbconfig/20191121-082108-marostegui.json [production]
07:57 <akosiaris> upgrade OTRS to 5.0.39 T225925 [production]
07:56 <marostegui> Promote db2133 to codfw m2 master - T238183 [production]
07:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1086 after upgrade', diff saved to https://phabricator.wikimedia.org/P9706 and previous config saved to /var/cache/conftool/dbconfig/20191121-072543-marostegui.json [production]