|
2017-06-08
§
|
| 09:05 |
<elukey> |
upgrade zookeeper packages to 3.4.5+dfsg-2+deb8u2 on conf100[123], conf200[23] and druid100[123] |
[production] |
| 08:58 |
<godog> |
swift eqiad-prod eqiad-prod: decom ms-be1005/6/7 - T166489 |
[production] |
| 08:50 |
<TabbyCat> |
Rename user "Mlpearc" to "FlightTime" on Central Auth is now finished (T166028) |
[production] |
| 08:36 |
<godog> |
temporarily stop ircecho on tegmen, puppet spam |
[production] |
| 08:21 |
<TabbyCat> |
Starting big global rename as requested in T166028 |
[production] |
| 07:00 |
<marostegui> |
Drop table updates on s6 - T139342 |
[production] |
| 05:59 |
<_joe_> |
uploading new service-checker version to reprepro, T167048 |
[production] |
| 05:54 |
<marostegui> |
Deploy alter table s2 - db1074 - T166205 |
[production] |
| 05:52 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1074 - T166205 (duration: 00m 43s) |
[production] |
| 05:43 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1076 - T166205 (duration: 00m 45s) |
[production] |
| 02:56 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Thu Jun 8 02:56:27 UTC 2017 (duration 6m 26s) |
[production] |
| 02:50 |
<l10nupdate@tin> |
scap sync-l10n completed (1.30.0-wmf.4) (duration: 05m 07s) |
[production] |
| 02:40 |
<twentyafterfour> |
deploying hotfix for T166958 |
[production] |
| 02:34 |
<l10nupdate@tin> |
scap sync-l10n completed (1.30.0-wmf.2) (duration: 08m 41s) |
[production] |
| 01:45 |
<mutante> |
manually running mediawiki maintenance job "echo_mail_batch" (on terbium as www-data, just like cron). did _NOT_ get denied by DB (T167373) |
[production] |
| 01:37 |
<maxsem@tin> |
Synchronized php-1.30.0-wmf.2/extensions/GeoData/includes/Searcher.php: Livehack to stop exceptions (duration: 00m 46s) |
[production] |
| 00:54 |
<mutante> |
cp4019 - powercycled (same as others) | lvs1007 - sits at installer - waiting for IP to be configured (T150256) |
[production] |
| 00:47 |
<mutante> |
cp1059 - same thing - powercycle after failed boot after reimaging script |
[production] |
| 00:41 |
<mutante> |
cp4011 - like cp4010 - powercycling (host down, console sat at initramfs). it hat the "did not detect disk by uid" issue but boots normal after powercycle |
[production] |
| 00:34 |
<mutante> |
cp4020 - powercycling (host down, console sat at initramfs) |
[production] |
| 00:31 |
<mutante> |
cp2012 - fixed salt key issue as for cp3005 (delete key, stop/start minion, accept new key) |
[production] |
| 00:25 |
<mutante> |
salt-master: deleted salt-key for cp3005, stopped started minion cp3005 - key got accepted again (was: Salt Master has rejected this minion's public key) |
[production] |
|
2017-06-07
§
|
| 23:33 |
<ppchelko@tin> |
Finished deploy [trending-edits/deploy@e0a8716]: Include reverts from bots to get rid of false positives (duration: 07m 00s) |
[production] |
| 23:30 |
<catrope@tin> |
Synchronized php-1.30.0-wmf.4/extensions/RelatedArticles/resources/ext.relatedArticles.readMore.eventLogging/index.js: T167236 (duration: 00m 43s) |
[production] |
| 23:28 |
<catrope@tin> |
Synchronized wmf-config/InitialiseSettings.php: Relaunch related pages A/B test to 98% of users on enwiki (T167310) (duration: 00m 44s) |
[production] |
| 23:26 |
<ppchelko@tin> |
Started deploy [trending-edits/deploy@e0a8716]: Include reverts from bots to get rid of false positives |
[production] |
| 22:24 |
<bblack> |
reimaging ex-cache_maps hosts (fresh role::spare::system installs) |
[production] |
| 22:18 |
<bblack> |
puppet node clean+deactivate for cp3003 |
[production] |
| 22:15 |
<bblack> |
lvs4002 - restarting pybal to remove old maps table entries |
[production] |
| 22:14 |
<bblack> |
lvs3002 - restarting pybal to remove old maps table entries |
[production] |
| 22:13 |
<bblack> |
lvs2002 - restarting pybal to remove old maps table entries |
[production] |
| 22:13 |
<bblack> |
lvs1002 - restarting pybal to remove old maps table entries |
[production] |
| 22:12 |
<bblack> |
lvs4004 - restarting pybal to remove old maps table entries |
[production] |
| 22:11 |
<bblack> |
lvs3004 - restarting pybal to remove old maps table entries |
[production] |
| 22:09 |
<bblack> |
lvs2005 - restarting pybal to remove old maps table entries |
[production] |
| 22:07 |
<bblack> |
lvs1005 - restarting pybal to remove old maps table entries |
[production] |
| 21:32 |
<twentyafterfour@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.2 |
[production] |
| 21:31 |
<twentyafterfour> |
rolling back to wmf.2 due to error spike and popups no longer working refs T166829 |
[production] |
| 21:25 |
<twentyafterfour@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.4 |
[production] |
| 21:23 |
<twentyafterfour@tin> |
Synchronized php-1.30.0-wmf.4/: sync 3248a17f5529ab6d5cbabb75dd0e72a7589f8633 refs T167343 (duration: 07m 52s) |
[production] |
| 20:26 |
<twentyafterfour@tin> |
Synchronized php-1.30.0-wmf.4/extensions/MobileFrontend: Deploy 66ef9cbd7f3de2832154f97392c2418fb1cd56ec refs T167216 (duration: 00m 46s) |
[production] |
| 20:04 |
<twentyafterfour> |
Preparing to deploy the MediaWiki train for group1 wikis, 1.30.0-wmf.4 refs T166829 |
[production] |
| 18:22 |
<thcipriani@tin> |
Synchronized wmf-config: SWAT: [[gerrit:357236|Enable archive indexing on delete for select wikis]] T162302 (duration: 00m 47s) |
[production] |
| 18:14 |
<thcipriani@tin> |
Synchronized portals: SWAT: [[gerrit:357490|Updating portals stats]] T128546 (duration: 00m 44s) |
[production] |
| 18:13 |
<thcipriani@tin> |
Synchronized portals/prod/wikipedia.org/assets: SWAT: [[gerrit:357490|Updating portals stats]] T128546 (duration: 00m 44s) |
[production] |
| 17:14 |
<elukey> |
restart nutcracker on thumbor1002 (too many connections approaching the 1024 ulimit) |
[production] |
| 15:37 |
<akosiaris> |
disable puppet on puppetmaster1001, depool rhodium for tests |
[production] |
| 14:51 |
<filippo@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=ms-fe2007.codfw.wmnet |
[production] |
| 14:48 |
<filippo@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=ms-fe1007.codfw.wmnet |
[production] |
| 14:11 |
<dcausse> |
eu swat done |
[production] |