|
2019-05-23
ยง
|
| 10:28 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:16 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Fully repool db1080 (duration: 00m 57s) |
[production] |
| 10:15 |
<_joe_> |
restarted php7.2-fpm on mw1261 to assess the effect of a larger APCu shm size T223180 |
[production] |
| 10:00 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:00 |
<moritzm> |
rebooting remaining mw servers in codfw (sans mcrouter proxies for now) |
[production] |
| 10:00 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:51 |
<hashar@deploy1001> |
Synchronized php-1.34.0-wmf.6/extensions/Collection: Rename wfAjaxCollectionGetItemList() T224093 (duration: 00m 57s) |
[production] |
| 09:37 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1080 into API (duration: 00m 55s) |
[production] |
| 09:22 |
<godog> |
bounce rsyslog on lithium - listener stuck /T199406 |
[production] |
| 09:10 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:10 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:10 |
<moritzm> |
rebooting scb servers in eqiad |
[production] |
| 09:02 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Slowly repool db1080 (duration: 00m 55s) |
[production] |
| 08:49 |
<hashar@deploy1001> |
rebuilt and synchronized wikiversions files: Re apply group1 wikis to 1.34.0-wmf.6 T220731 |
[production] |
| 08:29 |
<marostegui> |
Upgrade MySQL and kernel on db1080 |
[production] |
| 08:29 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1080 (duration: 00m 55s) |
[production] |
| 08:26 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:26 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:26 |
<moritzm> |
rebooting scb servers in codfw |
[production] |
| 07:58 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Slowly repool db1080 (duration: 00m 56s) |
[production] |
| 07:34 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:34 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:33 |
<moritzm> |
rebooting swift frontends in eqiad |
[production] |
| 07:26 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: More traffic to db1136 (duration: 00m 53s) |
[production] |
| 07:11 |
<marostegui> |
Stop MySQL on db1117:3323 to clone db1128 T222682 |
[production] |
| 06:38 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: More traffic to db1136 (duration: 00m 55s) |
[production] |
| 06:29 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove db2065 from config as it will be moved to m3 to replace db2042 (duration: 00m 55s) |
[production] |
| 06:28 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Remove db2065 from config as it will be moved to m3 to replace db2042 (duration: 00m 56s) |
[production] |
| 06:14 |
<mobrovac> |
start ruwiki dumps to fill the new parsoid tables - T215956 |
[production] |
| 05:33 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Promote db2070 as m5 codfw master - T221533 (duration: 00m 54s) |
[production] |
| 05:29 |
<marostegui> |
Promote db2070 to m5 codfw master instead of db2037 - T221533 |
[production] |
| 05:20 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Clarify db2107 status - will be the new master (duration: 00m 54s) |
[production] |
| 05:03 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Pool db1136 into s7 T222682 (duration: 00m 55s) |
[production] |
| 05:02 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Pool db1136 into s7 T222682 (duration: 00m 55s) |
[production] |
| 04:57 |
<mobrovac> |
decommission restbase1009-a - T223976 |
[production] |
| 04:46 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1080 (duration: 00m 55s) |
[production] |
| 04:40 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1080 (duration: 00m 58s) |
[production] |
| 04:24 |
<mobrovac> |
start nl, pt, pl wiki dumps to fill the new parsoid tables - T215956 |
[production] |
| 03:50 |
<twentyafterfour> |
m3 database activity levels look like they have returned to normal |
[production] |
| 03:48 |
<twentyafterfour> |
puppet runs cleanly on phab1003 |
[production] |
| 03:39 |
<mutante> |
phab1003 - disabling puppet; /etc/php/7.2/fpm/conf.d# ln -s /etc/php/7.2/mods-available/ldap.ini 20-ldap.ini ; systemctl restart php7.2-fpm |
[production] |
| 03:27 |
<twentyafterfour> |
restarted php-fpm on phab1003 |
[production] |
| 02:56 |
<mutante> |
phab1001 - removing community_metrics and project_changes cron jobs to avoid duplicate mails |
[production] |
| 02:51 |
<mutante> |
phab1003 - chown -R phd /srv/repos/ |
[production] |
| 02:41 |
<twentyafterfour> |
downtimed the systemd state on phab1001 for 1 year |
[production] |
| 02:35 |
<mutante> |
phabricator - going read-write again |
[production] |
| 02:24 |
<twentyafterfour> |
manually started aphlict on phab1003 |
[production] |
| 02:06 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=phab1003-vcs.eqiad.wmnet |
[production] |
| 02:04 |
<mutante> |
puppetmaster1001 - sudo -i conftool-merge |
[production] |
| 01:52 |
<twentyafterfour> |
phabricator is now served by phab1003 though still in read-only mode for a bit longer |
[production] |