|
2016-09-16
§
|
| 15:18 |
<akosiaris> |
enable puppet on puppetmaster1001 again |
[production] |
| 15:03 |
<akosiaris> |
disabling puppet on puppetmaster1001 |
[production] |
| 15:00 |
<hashar> |
gallium: dpkg --purge php5-mysql (mysql got removed) |
[production] |
| 14:46 |
<gehel> |
disabling shard allocation check on relforge to test shard allocation issues |
[production] |
| 13:56 |
<elukey> |
mw1189 back serving traffic after reimage |
[production] |
| 13:41 |
<akosiaris@tin> |
Synchronized wmf-config/db-eqiad.php: (no message) (duration: 00m 46s) |
[production] |
| 13:35 |
<hashar> |
gallium: removing MySQL which is no more defined in puppet and running puppet. Did: apt-get remove mysql-common mysql-server mysql-server-core-5.5 |
[production] |
| 13:19 |
<akosiaris@tin> |
Synchronized wmf-config/db-eqiad.php: (no message) (duration: 00m 48s) |
[production] |
| 12:50 |
<hashar@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: All wikis back to 1.28.0-wmf.18 :( T145819 |
[production] |
| 12:40 |
<hashar> |
Going to rollback all Wikis back to 1.28.0-wmf.18 . Despite much investigation, a bunch of jobs are broken due to T145819 which includes Special:CreateAccount :( |
[production] |
| 12:37 |
<elukey> |
mw1190 back serving traffic after the reimage |
[production] |
| 12:24 |
<gehel> |
rolling restart of codfw elasticsearch cluster completed - T145404 |
[production] |
| 12:20 |
<moritzm> |
installing security updates for mysql 5.5 (one off systems running mysql as packaged by Ubuntu/Debian and not running wmf-mariadb10) |
[production] |
| 12:15 |
<moritzm> |
installing python-imaging security updates on precise |
[production] |
| 11:24 |
<akosiaris> |
silence icinga-wm for a while |
[production] |
| 11:22 |
<akosiaris> |
restarted puppetmaster on all puppetmasters |
[production] |
| 11:22 |
<akosiaris> |
stop puppetmaster on all puppetmasters, resizing /var/lib/puppet |
[production] |
| 10:21 |
<marostegui> |
renaming tables before dropping them in codfw S1,S3,S4 - T54924 |
[production] |
| 09:34 |
<elukey> |
reimage mw1189-90 to Jessie (trying Riccardo's script!) |
[production] |
| 09:02 |
<moritzm> |
reimaging mw1252-mw1254 to jessie |
[production] |
| 08:59 |
<moritzm> |
installing tomcat7 security updates |
[production] |
| 08:52 |
<moritzm> |
installing tomcat8 security updates |
[production] |
| 08:43 |
<moritzm> |
installing libidn security updates in eqiad |
[production] |
| 07:36 |
<elukey> |
forced logrotation with debug of /etc/logrotate.d/graphite-web on graphite1001 to find cronspam source |
[production] |
| 03:21 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Fri Sep 16 03:21:54 UTC 2016 (duration 5m 18s) |
[production] |
| 03:16 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.19) (duration: 18m 44s) |
[production] |
| 02:41 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.18) (duration: 18m 25s) |
[production] |
| 01:28 |
<mutante> |
mw1294 - down and frozen, powercycled |
[production] |
| 00:53 |
<awight> |
remove "pending" from AMQ old message consumer |
[production] |
| 00:50 |
<awight> |
update paymentswiki config to disable legacy orphan rectifier |
[production] |
|
2016-09-15
§
|
| 23:38 |
<legoktm> |
legoktm@terbium:~$ foreachwiki extensions/WikimediaMaintenance/createExtensionTables.php babel # T145366 |
[production] |
| 23:24 |
<awight> |
rolled back paymentswiki to 392d67520d14998d61823755b22df50ab45afb35 |
[production] |
| 23:22 |
<awight> |
updating paymentswiki to possible broken 609c1e5f03ddcd9d72f562d7911cf77cc459371b |
[production] |
| 23:00 |
<awight> |
update paymentswiki to from 996ca30076946c6148d9688a905c82e2e346e165 to 392d67520d14998d61823755b22df50ab45afb35 (reverted DI submodule update) |
[production] |
| 22:55 |
<awight> |
Reenabled donations and fredge consumers |
[production] |
| 22:50 |
<awight> |
update fundraising CRM from f381bd1c15e30e4d47fe372e128143baee6a7c7a to 5ba6976f2552564b51085abff0afd5f76195229b |
[production] |
| 22:29 |
<awight> |
rollback paymentswiki from 609c1e5f03ddcd9d72f562d7911cf77cc459371b to 996ca30076946c6148d9688a905c82e2e346e165 |
[production] |
| 22:28 |
<awight> |
update paymentswiki from 996ca30076946c6148d9688a905c82e2e346e165 to 609c1e5f03ddcd9d72f562d7911cf77cc459371b |
[production] |
| 22:19 |
<ejegg> |
updated SmashPig from af19422065c08669269179019fea1e7c208d8a7e to db68be988194c960aebca691d0fd8e6a6d24246a |
[production] |
| 22:10 |
<ejegg> |
updated SmashPig from 12a7b78cf280f96ff3d2314abb2a9290506ae454 to af19422065c08669269179019fea1e7c208d8a7e |
[production] |
| 21:41 |
<ejegg> |
updated SmashPig from e11af5793df9a0e2dbcf9ac138a8f2a3fa1bf574 to 12a7b78cf280f96ff3d2314abb2a9290506ae454 |
[production] |
| 20:55 |
<hashar> |
All wikis are on 1.28.0-wmf.19 wikidatawiki / testwikidatawiki stick to .18 for now. |
[production] |
| 20:35 |
<gehel> |
increasing number of shards per node for dewiki_content index to 2 on elasticsearch codfw |
[production] |
| 20:27 |
<hashar@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: All wiki to .19. Keep testwikidata and wikidata at .18 (commits: 38603f0 770d336) |
[production] |
| 20:14 |
<yuvipanda> |
remove self from github wikimedia org, was getting spammed for each new repo creation |
[production] |
| 20:13 |
<ema> |
varnish-be esams cache_upload: rolling depool and restart |
[production] |
| 20:11 |
<ejegg> |
updated SmashPig from e11af5793df9a0e2dbcf9ac138a8f2a3fa1bf574 to 12a7b78cf280f96ff3d2314abb2a9290506ae454 |
[production] |
| 20:08 |
<gehel> |
increasing number of shards per node for enwiki_content index to 2 on elasticsearch codfw |
[production] |
| 20:01 |
<yuvipanda> |
restart puppetmaster on labcontrol1001 to pick up hiera changes |
[production] |
| 19:59 |
<ema> |
depool and restart varnish-be on cp1048 |
[production] |