|
2016-03-23
§
|
| 14:33 |
<godog> |
rolling-restart restbase after https://gerrit.wikimedia.org/r/279112 |
[production] |
| 12:29 |
<godog> |
pool restbase1012 / restbase1013 |
[production] |
| 12:27 |
<godog> |
halt restbase1003 / restbase1004 |
[production] |
| 12:17 |
<moritzm> |
installing various security updates on mediawiki eqiad servers (along with HHVM restarts): graphite2, libldap, pixman, sqlite, pygments, gnutls26 (already running fine on canaries since yesterday) |
[production] |
| 11:54 |
<godog> |
swift eqiad-prod ms-be1020 / ms-be1021 to weight 3500 |
[production] |
| 10:59 |
<jynus> |
stopping and restarting db1015 for upgrade and clone to db1077 |
[production] |
| 10:44 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1015, increase weight of db1044 (duration: 00m 25s) |
[production] |
| 10:38 |
<godog> |
depool restbase1003 / restbase1004 prior to deprovisioning the hardware |
[production] |
| 10:29 |
<moritzm> |
installing various security updates on mediawiki codfw servers (along with HHVM restarts): graphite2, libldap, pixman, sqlite, pygments, gnutls26 (already running fine on canaries since yesterday) |
[production] |
| 09:21 |
<jynus> |
start mysql on es2019 at es2018-bin.000044:287914983 |
[production] |
| 08:38 |
<jynus> |
starting mysql at es2019 |
[production] |
| 08:37 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Add db2008 (x1) depooled, depool es2019 (duration: 00m 26s) |
[production] |
| 08:30 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1044, pool db1075 (duration: 00m 25s) |
[production] |
| 08:27 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Add db1075 (duration: 00m 40s) |
[production] |
| 07:59 |
<jynus> |
powercycling es2019 - it was down |
[production] |
| 07:19 |
<_joe_> |
progressively activating cross-dc replica and encryption between the jobqueue redises |
[production] |
| 03:34 |
<mutante> |
tin - re-arm keyholder |
[production] |
| 03:26 |
<mutante> |
tin - restart keyholder - re: < twentyafterfour> 1 mismatch. I guess that keyholder-proxy needs to be restarted on tin |
[production] |
| 03:12 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Wed Mar 23 03:12:07 UTC 2016 (duration 9m 20s) |
[production] |
| 03:02 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.18) (duration: 17m 31s) |
[production] |
| 02:27 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.17) (duration: 11m 18s) |
[production] |
|
2016-03-22
§
|
| 23:47 |
<maxsem@tin> |
Synchronized php-1.27.0-wmf.18/extensions/WikimediaEvents/: https://gerrit.wikimedia.org/r/#/c/279066/ (duration: 00m 27s) |
[production] |
| 23:46 |
<maxsem@tin> |
Synchronized php-1.27.0-wmf.17/extensions/WikimediaEvents/: https://gerrit.wikimedia.org/r/#/c/279066/ (duration: 00m 28s) |
[production] |
| 23:40 |
<maxsem@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT x 3 (duration: 00m 26s) |
[production] |
| 23:35 |
<maxsem@tin> |
Synchronized portals: (no message) (duration: 00m 25s) |
[production] |
| 23:35 |
<maxsem@tin> |
Synchronized portals/prod/wikipedia.org/assets: (no message) (duration: 00m 26s) |
[production] |
| 23:26 |
<MaxSem> |
ran echo 'https://test.wikidata.org/static/favicon/testwikidata.ico' | mwscript purgeList.php |
[production] |
| 23:23 |
<maxsem@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/278334/ (duration: 00m 27s) |
[production] |
| 23:23 |
<maxsem@tin> |
Synchronized static/images/project-logos: https://gerrit.wikimedia.org/r/#/c/278334/ (duration: 00m 26s) |
[production] |
| 23:20 |
<maxsem@tin> |
Synchronized static/favicon/testwikidata.ico: https://gerrit.wikimedia.org/r/#/c/278321/ (duration: 00m 31s) |
[production] |
| 23:15 |
<mutante> |
mira - restarting and re-arming keyholder |
[production] |
| 23:14 |
<mutante> |
tin - restarting and re-arming keyholder |
[production] |
| 23:04 |
<mutante> |
re-arm keyholder on tin for new phab key |
[production] |
| 23:02 |
<mutante> |
mira re-arm keyholder for new phab key |
[production] |
| 22:56 |
<aaron@tin> |
Synchronized wmf-config: Lower "max lag" and $wgAPIMaxLagThreshold to 8/6 (duration: 00m 29s) |
[production] |
| 22:52 |
<aaron@tin> |
Synchronized wmf-config/jobqueue-eqiad.php: Bump timeout to 300ms (duration: 00m 29s) |
[production] |
| 22:50 |
<aaron@tin> |
Synchronized wmf-config/filebackend-production.php: comment tweak (duration: 00m 35s) |
[production] |
| 22:50 |
<mutante> |
replacing phab deploy key with a new one |
[production] |
| 21:41 |
<hoo> |
Ran sync-common on mw1139 (which missed two deploys) |
[production] |
| 21:28 |
<bd808> |
Restarted logstash process on logstash1002; dead from OOM since 2016-03-18T11:47:12 |
[production] |
| 21:27 |
<hoo@tin> |
Synchronized php-1.27.0-wmf.18/extensions/Wikidata: Update Wikibase: Fix add qualifier link not getting disabled (duration: 03m 43s) |
[production] |
| 21:23 |
<hoo@tin> |
Synchronized php-1.27.0-wmf.17/extensions/Wikidata: Update Wikibase: Fix add qualifier link not getting disabled (duration: 07m 30s) |
[production] |
| 21:01 |
<mutante> |
mira - arm keyholder |
[production] |
| 20:59 |
<mutante> |
tin - arm keyholder with deployment keys |
[production] |
| 20:47 |
<urandom> |
Stopping restbase2004.codfw.wmnet for offline sstablescrub : T130254 |
[production] |
| 20:46 |
<urandom> |
decommissioning restbase1004-a.eqiad.wmnet : T125842 |
[production] |
| 20:45 |
<urandom> |
Rolling restart of RESTBase Cassandra cluster complete : T130393, T128787 |
[production] |
| 20:25 |
<thcipriani@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.27.0-wmf.18 |
[production] |
| 20:22 |
<gehel> |
deployed latest version of WDQS |
[production] |
| 20:18 |
<thcipriani@tin> |
Finished scap: testwiki to php-1.27.0-wmf.18 and rebuild l10n cache (duration: 46m 09s) |
[production] |