9301-9350 of 10000 results (58ms)
2016-09-16 §
15:03 <akosiaris> disabling puppet on puppetmaster1001 [production]
15:00 <hashar> gallium: dpkg --purge php5-mysql (mysql got removed) [production]
14:46 <gehel> disabling shard allocation check on relforge to test shard allocation issues [production]
13:56 <elukey> mw1189 back serving traffic after reimage [production]
13:41 <akosiaris@tin> Synchronized wmf-config/db-eqiad.php: (no message) (duration: 00m 46s) [production]
13:35 <hashar> gallium: removing MySQL which is no more defined in puppet and running puppet. Did: apt-get remove mysql-common mysql-server mysql-server-core-5.5 [production]
13:19 <akosiaris@tin> Synchronized wmf-config/db-eqiad.php: (no message) (duration: 00m 48s) [production]
12:50 <hashar@tin> rebuilt wikiversions.php and synchronized wikiversions files: All wikis back to 1.28.0-wmf.18 :( T145819 [production]
12:40 <hashar> Going to rollback all Wikis back to 1.28.0-wmf.18 . Despite much investigation, a bunch of jobs are broken due to T145819 which includes Special:CreateAccount :( [production]
12:37 <elukey> mw1190 back serving traffic after the reimage [production]
12:24 <gehel> rolling restart of codfw elasticsearch cluster completed - T145404 [production]
12:20 <moritzm> installing security updates for mysql 5.5 (one off systems running mysql as packaged by Ubuntu/Debian and not running wmf-mariadb10) [production]
12:15 <moritzm> installing python-imaging security updates on precise [production]
11:24 <akosiaris> silence icinga-wm for a while [production]
11:22 <akosiaris> restarted puppetmaster on all puppetmasters [production]
11:22 <akosiaris> stop puppetmaster on all puppetmasters, resizing /var/lib/puppet [production]
10:21 <marostegui> renaming tables before dropping them in codfw S1,S3,S4 - T54924 [production]
09:34 <elukey> reimage mw1189-90 to Jessie (trying Riccardo's script!) [production]
09:02 <moritzm> reimaging mw1252-mw1254 to jessie [production]
08:59 <moritzm> installing tomcat7 security updates [production]
08:52 <moritzm> installing tomcat8 security updates [production]
08:43 <moritzm> installing libidn security updates in eqiad [production]
07:36 <elukey> forced logrotation with debug of /etc/logrotate.d/graphite-web on graphite1001 to find cronspam source [production]
03:21 <l10nupdate@tin> ResourceLoader cache refresh completed at Fri Sep 16 03:21:54 UTC 2016 (duration 5m 18s) [production]
03:16 <mwdeploy@tin> scap sync-l10n completed (1.28.0-wmf.19) (duration: 18m 44s) [production]
02:41 <mwdeploy@tin> scap sync-l10n completed (1.28.0-wmf.18) (duration: 18m 25s) [production]
01:28 <mutante> mw1294 - down and frozen, powercycled [production]
00:53 <awight> remove "pending" from AMQ old message consumer [production]
00:50 <awight> update paymentswiki config to disable legacy orphan rectifier [production]
2016-09-15 §
23:38 <legoktm> legoktm@terbium:~$ foreachwiki extensions/WikimediaMaintenance/createExtensionTables.php babel # T145366 [production]
23:24 <awight> rolled back paymentswiki to 392d67520d14998d61823755b22df50ab45afb35 [production]
23:22 <awight> updating paymentswiki to possible broken 609c1e5f03ddcd9d72f562d7911cf77cc459371b [production]
23:00 <awight> update paymentswiki to from 996ca30076946c6148d9688a905c82e2e346e165 to 392d67520d14998d61823755b22df50ab45afb35 (reverted DI submodule update) [production]
22:55 <awight> Reenabled donations and fredge consumers [production]
22:50 <awight> update fundraising CRM from f381bd1c15e30e4d47fe372e128143baee6a7c7a to 5ba6976f2552564b51085abff0afd5f76195229b [production]
22:29 <awight> rollback paymentswiki from 609c1e5f03ddcd9d72f562d7911cf77cc459371b to 996ca30076946c6148d9688a905c82e2e346e165 [production]
22:28 <awight> update paymentswiki from 996ca30076946c6148d9688a905c82e2e346e165 to 609c1e5f03ddcd9d72f562d7911cf77cc459371b [production]
22:19 <ejegg> updated SmashPig from af19422065c08669269179019fea1e7c208d8a7e to db68be988194c960aebca691d0fd8e6a6d24246a [production]
22:10 <ejegg> updated SmashPig from 12a7b78cf280f96ff3d2314abb2a9290506ae454 to af19422065c08669269179019fea1e7c208d8a7e [production]
21:41 <ejegg> updated SmashPig from e11af5793df9a0e2dbcf9ac138a8f2a3fa1bf574 to 12a7b78cf280f96ff3d2314abb2a9290506ae454 [production]
20:55 <hashar> All wikis are on 1.28.0-wmf.19 wikidatawiki / testwikidatawiki stick to .18 for now. [production]
20:35 <gehel> increasing number of shards per node for dewiki_content index to 2 on elasticsearch codfw [production]
20:27 <hashar@tin> rebuilt wikiversions.php and synchronized wikiversions files: All wiki to .19. Keep testwikidata and wikidata at .18 (commits: 38603f0 770d336) [production]
20:14 <yuvipanda> remove self from github wikimedia org, was getting spammed for each new repo creation [production]
20:13 <ema> varnish-be esams cache_upload: rolling depool and restart [production]
20:11 <ejegg> updated SmashPig from e11af5793df9a0e2dbcf9ac138a8f2a3fa1bf574 to 12a7b78cf280f96ff3d2314abb2a9290506ae454 [production]
20:08 <gehel> increasing number of shards per node for enwiki_content index to 2 on elasticsearch codfw [production]
20:01 <yuvipanda> restart puppetmaster on labcontrol1001 to pick up hiera changes [production]
19:59 <ema> depool and restart varnish-be on cp1048 [production]
19:56 <ema> depool and restart varnish-be on cp1064 [production]