3251-3300 of 10000 results (95ms)
2019-05-23 ยง
10:00 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
09:51 <hashar@deploy1001> Synchronized php-1.34.0-wmf.6/extensions/Collection: Rename wfAjaxCollectionGetItemList() T224093 (duration: 00m 57s) [production]
09:37 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1080 into API (duration: 00m 55s) [production]
09:22 <godog> bounce rsyslog on lithium - listener stuck /T199406 [production]
09:10 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:10 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
09:10 <moritzm> rebooting scb servers in eqiad [production]
09:02 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1080 (duration: 00m 55s) [production]
08:49 <hashar@deploy1001> rebuilt and synchronized wikiversions files: Re apply group1 wikis to 1.34.0-wmf.6 T220731 [production]
08:29 <marostegui> Upgrade MySQL and kernel on db1080 [production]
08:29 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1080 (duration: 00m 55s) [production]
08:26 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:26 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
08:26 <moritzm> rebooting scb servers in codfw [production]
07:58 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1080 (duration: 00m 56s) [production]
07:34 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
07:34 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
07:33 <moritzm> rebooting swift frontends in eqiad [production]
07:26 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1136 (duration: 00m 53s) [production]
07:11 <marostegui> Stop MySQL on db1117:3323 to clone db1128 T222682 [production]
06:38 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1136 (duration: 00m 55s) [production]
06:29 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db2065 from config as it will be moved to m3 to replace db2042 (duration: 00m 55s) [production]
06:28 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db2065 from config as it will be moved to m3 to replace db2042 (duration: 00m 56s) [production]
06:14 <mobrovac> start ruwiki dumps to fill the new parsoid tables - T215956 [production]
05:33 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Promote db2070 as m5 codfw master - T221533 (duration: 00m 54s) [production]
05:29 <marostegui> Promote db2070 to m5 codfw master instead of db2037 - T221533 [production]
05:20 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Clarify db2107 status - will be the new master (duration: 00m 54s) [production]
05:03 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Pool db1136 into s7 T222682 (duration: 00m 55s) [production]
05:02 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool db1136 into s7 T222682 (duration: 00m 55s) [production]
04:57 <mobrovac> decommission restbase1009-a - T223976 [production]
04:46 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1080 (duration: 00m 55s) [production]
04:40 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1080 (duration: 00m 58s) [production]
04:24 <mobrovac> start nl, pt, pl wiki dumps to fill the new parsoid tables - T215956 [production]
03:50 <twentyafterfour> m3 database activity levels look like they have returned to normal [production]
03:48 <twentyafterfour> puppet runs cleanly on phab1003 [production]
03:39 <mutante> phab1003 - disabling puppet; /etc/php/7.2/fpm/conf.d# ln -s /etc/php/7.2/mods-available/ldap.ini 20-ldap.ini ; systemctl restart php7.2-fpm [production]
03:27 <twentyafterfour> restarted php-fpm on phab1003 [production]
02:56 <mutante> phab1001 - removing community_metrics and project_changes cron jobs to avoid duplicate mails [production]
02:51 <mutante> phab1003 - chown -R phd /srv/repos/ [production]
02:41 <twentyafterfour> downtimed the systemd state on phab1001 for 1 year [production]
02:35 <mutante> phabricator - going read-write again [production]
02:24 <twentyafterfour> manually started aphlict on phab1003 [production]
02:06 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=phab1003-vcs.eqiad.wmnet [production]
02:04 <mutante> puppetmaster1001 - sudo -i conftool-merge [production]
01:52 <twentyafterfour> phabricator is now served by phab1003 though still in read-only mode for a bit longer [production]
01:52 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=phab1003-vcs.eqiad.wmnet [production]
01:49 <mutante> puppetmaster1001 - conftool-merge [production]
01:41 <eileen> civicrm revision changed from e6e846708f to 21afd001b6, config revision is 87e78d3eac [production]
01:37 <mutante> depooled phab1001-vcs from git-ssh via conftool [production]
01:36 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=phab1001-vcs.eqiad.wmnet [production]