|
2017-05-29
§
|
| 12:52 |
<akosiaris> |
enable puppet across codfw/ulsfo after puppetmaster upgrade |
[production] |
| 12:41 |
<akosiaris> |
disable puppet across codfw/ulsfo for puppetmaster upgrade. This should avoid any irc spam about failed puppet agent runs |
[production] |
| 12:36 |
<moritzm> |
installing imagemagick security updates on jessie |
[production] |
| 12:31 |
<akosiaris> |
update kubernetes policy-options on cr{1,2}-{eqiad,codfw}. T165732 |
[production] |
| 10:39 |
<moritzm> |
installing fop security updates |
[production] |
| 10:18 |
<ema> |
upgrade nginx to 1.11.10-1+wmf1 on hassium and hassaleh |
[production] |
| 09:53 |
<moritzm> |
upgrade remaining mw* hosts already running HHVM 3.18 to 3.18.2+dfsg-1+wmf4 |
[production] |
| 09:22 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1045 (duration: 00m 41s) |
[production] |
| 09:01 |
<marostegui> |
Drop gather tables from: testwiki, test2wiki, enwikivoyage, hewiki, enwiki - T166097 |
[production] |
| 08:02 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Remove db1023 - T166486 (duration: 00m 41s) |
[production] |
| 08:02 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Remove db1023 - T166486 (duration: 00m 42s) |
[production] |
| 07:38 |
<marostegui> |
Stop MySQL on db1095 to take a backup - this will make labsdb1009,10 and 11 break replication while it is down - T153743 |
[production] |
| 07:01 |
<_joe_> |
reeanbling scap on mw2140, T166328 |
[production] |
| 06:45 |
<_joe_> |
restarting changeprop on scb1002, using 15 gigs of RAM |
[production] |
| 06:42 |
<marostegui> |
Deploy alter table s3 - dbstore2002 - T166278 |
[production] |
| 06:41 |
<marostegui> |
Deploy alter table s4 - dbstore1002 - T166206 |
[production] |
| 06:33 |
<_joe_> |
trying to restart pdfrender on scb1002 |
[production] |
| 06:32 |
<marostegui> |
Deploy alter table s3 - db2036 - T166278 |
[production] |
| 06:32 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2043, depool db2036 - T166278 (duration: 01m 44s) |
[production] |
| 06:29 |
<_joe_> |
powercycling mw1294 |
[production] |
| 06:11 |
<marostegui> |
Deploy alter table on s4 db1084 - T166206 |
[production] |
| 06:10 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1091, depool db1084 - T166206 (duration: 02m 45s) |
[production] |
| 06:01 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1091 - T166206 (duration: 03m 01s) |
[production] |
| 05:54 |
<marostegui> |
Restart MySQL on db1047 - T166452 |
[production] |
| 02:24 |
<l10nupdate@tin> |
scap sync-l10n completed (1.30.0-wmf.1) (duration: 08m 20s) |
[production] |
|
2017-05-26
§
|
| 14:29 |
<marostegui> |
Stop pt-table-checksum on s1 - T162807 |
[production] |
| 14:04 |
<marostegui> |
Deploy alter tabel on s3 revision table db2043 - T166278 |
[production] |
| 14:03 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2050, depool db2043 - T166278 (duration: 00m 41s) |
[production] |
| 13:57 |
<_joe_> |
consuming the backlog of htmlCacheUpdate jobs for enwiktionary |
[production] |
| 13:19 |
<gehel> |
restart wdqs-updater on all wdqs nodes - T166378 |
[production] |
| 12:55 |
<marostegui> |
Deploy alter table s4 on db1097 - T166206 |
[production] |
| 12:44 |
<elukey> |
Restart Hadoop daemons on analytics100[12] (Hadoop master nodes) for jvm upgrades |
[production] |
| 12:43 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1097 - T166206 (duration: 00m 41s) |
[production] |
| 10:56 |
<gehel> |
restart wdqs-updater on all wdqs nodes - T166378 |
[production] |
| 09:30 |
<volans> |
slowly testing if puppet stringify_facts=false is a noop across the fleet T166372 |
[production] |
| 08:45 |
<volans> |
killed daemonized puppet on tegmen, lvs1006 T166203 |
[production] |
| 06:38 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1097 - T166206 (duration: 00m 40s) |
[production] |
| 06:10 |
<marostegui> |
Deploy alter table on s3 - db2050 - T166278 |
[production] |
| 06:09 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2057, depool db2050 - T166278 (duration: 00m 56s) |
[production] |
| 06:05 |
<marostegui> |
Resume pt-table-checksum on s1 - T162807 |
[production] |
| 02:58 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Fri May 26 02:58:48 UTC 2017 (duration 6m 37s) |
[production] |
| 02:52 |
<l10nupdate@tin> |
scap sync-l10n completed (1.30.0-wmf.2) (duration: 07m 59s) |
[production] |
| 02:25 |
<l10nupdate@tin> |
scap sync-l10n completed (1.30.0-wmf.1) (duration: 08m 31s) |
[production] |
| 01:33 |
<bsitzmann@tin> |
Finished deploy [mobileapps/deploy@a8d0c91]: Update mobileapps to db6493c (duration: 03m 45s) |
[production] |
| 01:29 |
<bsitzmann@tin> |
Started deploy [mobileapps/deploy@a8d0c91]: Update mobileapps to db6493c |
[production] |