1501-1550 of 10000 results (22ms)
2020-09-21 ยง
21:28 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
21:23 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
21:18 <pt1979@cumin2001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:12 <pt1979@cumin2001> START - Cookbook sre.dns.netbox [production]
20:49 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: adjust enwiktionary completion search ranking (duration: 00m 57s) [production]
20:47 <ebernhardson@deploy1001> Synchronized php-1.36.0-wmf.9/extensions/CirrusSearch/: Remove pages from completion search by page id (duration: 01m 00s) [production]
20:04 <herron> moving prometheus instance from bast3004 to prometheus3001 T243057 [production]
19:46 <herron> moving prometheus instance from bast4002 to prometheus4001 T243057 [production]
19:38 <mholloway-shell@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Push notifications deployment (4/5) (duration: 00m 57s) [production]
19:34 <mholloway-shell@deploy1001> Synchronized wmf-config/CommonSettings.php: Push notifications deployment (3/5) (duration: 00m 57s) [production]
19:28 <mholloway-shell@deploy1001> Synchronized wmf-config/ProductionServices.php: Push notifications deployment (2/5) (duration: 00m 57s) [production]
19:26 <mholloway-shell@deploy1001> Synchronized wmf-config/LabsServices.php: Push notifications deployment (1/5) (duration: 00m 57s) [production]
19:19 <mholloway-shell@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
19:18 <mepps> updated crm to 8f32b6301f [production]
19:15 <mholloway-shell@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
19:14 <ejegg> updated fundraising CiviCRM from e5ebf9d18a to 8f32b6301f [production]
19:13 <mholloway-shell@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
18:59 <ppchelko@deploy1001> Synchronized wmf-config/CommonSettings.php: gerrit:622863 T249745 (duration: 00m 56s) [production]
18:57 <ebernhardson@deploy1001> Finished deploy [search/mjolnir/deploy@8afe8d2]: mjolnir daemons update I336365 (duration: 06m 54s) [production]
18:53 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable and configure GrowthExperiments on plwiki (T254239) and ptwiki (T255027) (duration: 00m 56s) [production]
18:50 <ebernhardson@deploy1001> Started deploy [search/mjolnir/deploy@8afe8d2]: mjolnir daemons update I336365 [production]
18:33 <mepps> updated crm from cc1f7e6d13 to e5ebf9d18a [production]
18:26 <catrope@deploy1001> Synchronized wmf-config/CommonSettings.php: Define Chinese logo variants for Modern Vector (no-op) (part 2) (T261153) (duration: 00m 56s) [production]
18:25 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Define Chinese logo variants for Modern Vector (no-op) (T261153) (duration: 00m 57s) [production]
18:21 <catrope@deploy1001> Synchronized static/images/mobile/copyright/: Update Chinese logo variants for Modern Vector (T261153) (duration: 00m 56s) [production]
18:08 <XioNoX> add NAT rule to pfw3-codfw - T263488 [production]
17:42 <papaul> rebooting ps1-a8-codfw firmware upgrade [production]
16:46 <papaul> shutting down ms-be2019 for BBU replacing [production]
16:24 <marostegui@cumin1001> dbctl commit (dc=all): 'db2127 (re)pooling @ 100%: Slowly repool after on-site maintenance T262247 ', diff saved to https://phabricator.wikimedia.org/P12696 and previous config saved to /var/cache/conftool/dbconfig/20200921-162433-root.json [production]
16:17 <papaul> replacing msw-c8-codfw [production]
16:16 <jayme@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:09 <marostegui@cumin1001> dbctl commit (dc=all): 'db2127 (re)pooling @ 75%: Slowly repool after on-site maintenance T262247 ', diff saved to https://phabricator.wikimedia.org/P12695 and previous config saved to /var/cache/conftool/dbconfig/20200921-160929-root.json [production]
16:08 <jayme@cumin1001> START - Cookbook sre.dns.netbox [production]
15:54 <marostegui@cumin1001> dbctl commit (dc=all): 'db2127 (re)pooling @ 50%: Slowly repool after on-site maintenance T262247 ', diff saved to https://phabricator.wikimedia.org/P12694 and previous config saved to /var/cache/conftool/dbconfig/20200921-155426-root.json [production]
15:51 <ladsgroup@deploy1001> Synchronized php-1.36.0-wmf.9/extensions/Wikibase/lib/includes/Store/Sql/Terms/: [[gerrit:628808|Introduce and use StatsdMonitoring trait in term store (T262923), Part I]] (duration: 00m 56s) [production]
15:50 <ladsgroup@deploy1001> Synchronized php-1.36.0-wmf.9/extensions/Wikibase/lib/includes/Store/Sql/Terms/Util/StatsdMonitoring.php: [[gerrit:628808|Introduce and use StatsdMonitoring trait in term store (T262923), Part I]] (duration: 00m 59s) [production]
15:44 <hnowlan@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) [production]
15:39 <marostegui@cumin1001> dbctl commit (dc=all): 'db2127 (re)pooling @ 25%: Slowly repool after on-site maintenance T262247 ', diff saved to https://phabricator.wikimedia.org/P12693 and previous config saved to /var/cache/conftool/dbconfig/20200921-153923-root.json [production]
15:24 <hnowlan> roll-restarting restbase-dev for java security updates [production]
15:24 <hnowlan@cumin1001> START - Cookbook sre.cassandra.roll-restart [production]
15:12 <kormat@cumin1001> dbctl commit (dc=all): 'Take db2124 back out of dump/vslow T259831', diff saved to https://phabricator.wikimedia.org/P12692 and previous config saved to /var/cache/conftool/dbconfig/20200921-151210-kormat.json [production]
15:10 <moritzm> rolling restart of mw canaries in codfw to pick up libx11 update [production]
15:07 <moritzm> installing libx11 security updates on stretch [production]
15:02 <kormat@cumin1001> dbctl commit (dc=all): 'db2117 (re)pooling @ 100%: schema change T259831', diff saved to https://phabricator.wikimedia.org/P12691 and previous config saved to /var/cache/conftool/dbconfig/20200921-150233-kormat.json [production]
14:47 <kormat@cumin1001> dbctl commit (dc=all): 'db2117 (re)pooling @ 75%: schema change T259831', diff saved to https://phabricator.wikimedia.org/P12690 and previous config saved to /var/cache/conftool/dbconfig/20200921-144729-kormat.json [production]
14:40 <moritzm> installing qemu security updates on ganeti* stretch nodes [production]
14:37 <papaul> firmware upgrade on db2127 [production]
14:36 <moritzm> installing qemu security updates on ganeti2011 and gnt-instance reboot debmonitor2001 [production]
14:36 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:36 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]