1801-1850 of 10000 results (26ms)
2020-06-15 ยง
12:01 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:59 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:57 <moritzm> reimaging sretest1002 to validate the reimage script on Buster [production]
11:43 <marostegui> Reimage dbproxy2003 which points to m3-master.codfw.wmnet (not in use) - T255408 [production]
11:40 <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:605543|GrowthExperiments: Switch on guidance feature (T239181)]] (duration: 00m 57s) [production]
11:10 <sukhe@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:10 <sukhe@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:07 <hnowlan> regenerated certificates for restbase2009, restbase101[678], restbase201[012]. Did not roll-restart yet [production]
11:07 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
11:04 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single [production]
11:03 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:03 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
10:54 <moritzm> imported python-phabricator 0.7.0-2~wmf2 to apt.wikimedia.org/buster-wikimedia T245114 [production]
10:39 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:605553| Bumping portals to master (605553)]] (duration: 00m 58s) [production]
10:38 <hnowlan> regenerated restbase2009's cassandra certificates [production]
10:38 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:605553| Bumping portals to master (605553)]] (duration: 00m 58s) [production]
10:16 <jmm@cumin1001> END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) [production]
10:16 <jmm@cumin1001> START - Cookbook sre.hosts.reboot-single [production]
10:12 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T254820 [enwikivoyage] Undeploy the Listings extension (duration: 01m 00s) [production]
10:08 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:05 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:53 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:50 <filippo@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:46 <godog> run logstash benchmark on logstash1023 [production]
09:42 <volans> deploying esams mgmt DNS records automatically generated by Netbox ( operations/dns/+/604136/ ) - T233183 [production]
09:41 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
09:35 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
09:29 <elukey> update analytics-in4/6 filters on cr1-cr2 eqiad to update the Druid term (new nodes added) [production]
09:21 <jbond42> offlining puppetmaster1003 and 2003 for reboot [production]
09:17 <XioNoX> reduce ae device-count from 10 to 3 on asw2-a/b/c-eqiad [production]
09:14 <jmm@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:11 <jmm@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:55 <marostegui> Deploy schema change on db2123 (s5 codfw master) - T250066 [production]
08:50 <kart_> Updated cxserver to 2020-06-10-044445-production (T246319, T254959) [production]
08:46 <kartik@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
08:42 <kartik@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
08:39 <kartik@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'cxserver' for release 'staging' . [production]
08:34 <moritzm> reimaging cumin2001 T245114 [production]
08:22 <marostegui> Switchover m3-master from dbproxy1008 to dbproxy1016 - T202367 [production]
08:17 <marostegui> Deploy schema change on db1131 (s6 master) - T250066 [production]
08:09 <moritzm> installing libexif security updates [production]
07:51 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
07:49 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
07:46 <XioNoX> standardize ae device-count on all routers [production]
07:36 <XioNoX> push new pfw firewall policies - T255185 [production]
07:28 <marostegui> Deploy schema change on db1093 [production]
07:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1093 for schema change', diff saved to https://phabricator.wikimedia.org/P11492 and previous config saved to /var/cache/conftool/dbconfig/20200615-072835-marostegui.json [production]
07:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2092', diff saved to https://phabricator.wikimedia.org/P11491 and previous config saved to /var/cache/conftool/dbconfig/20200615-072742-marostegui.json [production]
06:21 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
06:19 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]