|
2018-02-15
§
|
| 06:49 |
<twentyafterfour> |
starting phabricator upgrade tagged release/2018-02-15/1 |
[production] |
| 06:45 |
<twentyafterfour> |
restarted apache on phab1001 and reset cluster.read-only to false |
[production] |
| 06:44 |
<jynus> |
set db1059 in read-write |
[production] |
| 06:38 |
<jynus> |
merging dns update for phabricator db |
[production] |
| 06:35 |
<jynus> |
set db1043 as read only |
[production] |
| 06:34 |
<twentyafterfour> |
set cluster.read-only in phabricator |
[production] |
| 06:33 |
<jynus> |
about to set phabricator.wikimedia.org as read only |
[production] |
| 06:28 |
<jynus> |
scheduling downtime for phabricator on phab1001 |
[production] |
| 06:17 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Slowly repool db1110 (duration: 01m 13s) |
[production] |
| 06:06 |
<marostegui> |
Upgrade mysql on db1110 |
[production] |
| 05:57 |
<jynus> |
restarting dbproxy1008 for kernel upgrade |
[production] |
| 05:43 |
<andrew@tin> |
Finished deploy [horizon/deploy@c355366]: testing a couple of cherry-picks in horizon (duration: 03m 06s) |
[production] |
| 05:40 |
<andrew@tin> |
Started deploy [horizon/deploy@c355366]: testing a couple of cherry-picks in horizon |
[production] |
| 02:25 |
<l10nupdate@tin> |
scap sync-l10n completed (1.31.0-wmf.20) (duration: 07m 25s) |
[production] |
| 02:01 |
<mutante> |
phab1001 - restarted apache to fix server status page |
[production] |
| 01:27 |
<twentyafterfour> |
restarting apache2 on phab1001 to free deadlocked php processes. |
[production] |
| 01:03 |
<twentyafterfour> |
using the current phabricator maintenance window to deploy https://gerrit.wikimedia.org/r/#/c/410626/ |
[production] |
| 01:03 |
<twentyafterfour> |
the scheduled phabricator upgrade is delayed until 06:00 UTC Thursday because of large database migrations. Doing the upgrade at a time when DBAs are available to assist. |
[production] |
| 00:52 |
<maxsem@tin> |
Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/410267/ (duration: 01m 14s) |
[production] |
| 00:49 |
<maxsem@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/410267/ (duration: 01m 13s) |
[production] |
|
2018-02-14
§
|
| 23:39 |
<AaronSchulz> |
Running initSiteStats.php on s3 for T186947 |
[production] |
| 22:04 |
<aaron@tin> |
Synchronized php-1.31.0-wmf.20/includes/SiteStats.php: f549559dc0 (duration: 01m 13s) |
[production] |
| 21:52 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1088 with full weight (duration: 01m 13s) |
[production] |
| 21:38 |
<mholloway-shell@tin> |
Finished deploy [mobileapps/deploy@9bad612]: Update mobileapps to f23519f (duration: 06m 01s) |
[production] |
| 21:32 |
<mholloway-shell@tin> |
Started deploy [mobileapps/deploy@9bad612]: Update mobileapps to f23519f |
[production] |
| 21:30 |
<arlolra@tin> |
Finished deploy [parsoid/deploy@7961b3f]: Updating Parsoid to caee2ed (duration: 15m 12s) |
[production] |
| 21:15 |
<arlolra@tin> |
Started deploy [parsoid/deploy@7961b3f]: Updating Parsoid to caee2ed |
[production] |
| 21:00 |
<ema> |
upgrade cp1099 to varnish 5 (last upload@eqiad host) |
[production] |
| 20:54 |
<twentyafterfour@tin> |
Synchronized php-1.31.0-wmf.21/extensions/CentralNotice: Sync CentralNotice again after proper rebase (duration: 01m 14s) |
[production] |
| 20:43 |
<ema> |
upgrade cp1074 to varnish 5 |
[production] |
| 20:42 |
<twentyafterfour@tin> |
Synchronized php-1.31.0-wmf.21/extensions/CentralNotice/: sync https://gerrit.wikimedia.org/r/#/c/410346/ for Ejegg (duration: 01m 15s) |
[production] |
| 20:40 |
<twentyafterfour> |
Group1 wikis are now running MediaWiki 1.31.0-wmf.21 - still no blockers on T183960 |
[production] |
| 20:38 |
<twentyafterfour@tin> |
Synchronized php: group1 wikis to 1.31.0-wmf.21 (duration: 01m 12s) |
[production] |
| 20:37 |
<twentyafterfour@tin> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.31.0-wmf.21 |
[production] |
| 20:33 |
<ema> |
upgrade cp1073 to varnish 5 |
[production] |
| 20:05 |
<ema> |
upgrade cp1072 to varnish 5 |
[production] |
| 19:44 |
<ema> |
upgrade cp1071 to varnish 5 |
[production] |
| 19:25 |
<XioNoX> |
enabling netflow on cr1-eqiad |
[production] |
| 19:24 |
<no_justification> |
ran namespaceDupes.php --fix for hiwiki |
[production] |
| 19:24 |
<demon@tin> |
Synchronized wmf-config/InitialiseSettings.php: portal aliases for hiwiki (duration: 01m 13s) |
[production] |
| 19:22 |
<ema> |
upgrade cp1064 to varnish 5 |
[production] |
| 19:20 |
<no_justification> |
running updateCollation.php on nowikimedia |
[production] |
| 19:19 |
<demon@tin> |
Synchronized wmf-config/InitialiseSettings.php: nowikimedia collation, T185630 (duration: 01m 13s) |
[production] |
| 19:16 |
<andrewbogott> |
rebooting labvirt1019 so I can have a look at the raid setup, for T172538 |
[production] |
| 19:14 |
<no_justification> |
ran namespaceDupes.php --fix on wawiktionary |
[production] |
| 19:13 |
<demon@tin> |
Synchronized wmf-config/InitialiseSettings.php: wawiktionary namespaces, T185289 (duration: 01m 13s) |
[production] |
| 19:11 |
<demon@tin> |
Synchronized wmf-config/InitialiseSettings.php: Revert prior, busted the canaries (duration: 01m 15s) |
[production] |
| 19:08 |
<demon@tin> |
scap failed: average error rate on 7/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details) |
[production] |
| 19:06 |
<demon@tin> |
rebuilt and synchronized wikiversions files: namespace aliases for zhwiki, T184866 |
[production] |
| 19:00 |
<ema> |
upgrade cp1063 to varnish 5 |
[production] |