2401-2450 of 10000 results (33ms)
2015-12-07 §
18:36 <jynus> es1019 and its management interface became unresponsive [production]
18:16 <mutante> deleting pmtpa host groups from nagios/icinga [production]
17:04 <thcipriani@tin> Synchronized php-1.27.0-wmf.7/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android Beta app: 10% [[gerrit:254045]] (duration: 00m 29s) [production]
17:02 <thcipriani> reset uncommited change to includes/jobqueue/JobRunner.php in /srv/mediawiki-staging/php-1.27.0-wmf.7 [production]
17:00 <hashar> Restarted Jenkins to start the ZeroMQ publisher https://phabricator.wikimedia.org/T120668 [production]
16:56 <_joe_> upgrading pybal on primaries in esams, ulsfo, codfw [production]
16:44 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration for en.wiktionary [[gerrit:254476]] and Enable subpages on custom aliases from 112 to 119 [[gerrit:254475]] (duration: 00m 27s) [production]
16:43 <_joe_> installing pybal on eqiad backups [production]
16:43 <jynus> restarting mysql, upgrading and rebooting mysql on es1019 [production]
16:36 <_joe_> installing the new pybal version on the backup lvss in esams and ulsfo [production]
16:31 <_joe_> upgrading pybal on all the backup LVSs in codfw [production]
16:28 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: Rights configuration on fa.wikipedia [[gerrit:254486]] (duration: 00m 29s) [production]
16:28 <_joe_> upgrading pybal on lvs1007-12 [production]
16:19 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration for ur.wikipedia [[gerrit:254665]] (duration: 00m 30s) [production]
16:18 <_joe_> uploaded pybal 1.13.2 to reprepro [production]
16:14 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: Translate project namesapce for jbowiki [[gerrit:255954]] (duration: 00m 28s) [production]
15:39 <moritzm> uploaded openssl 1.0.2e-1~wmf1 to carbon [production]
15:37 <yurik> deployed kartotherian [production]
15:25 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Depool es1019; es1017 at 100% load; pool es1015 with low weight (duration: 00m 28s) [production]
14:56 <yurik> deployed latest tilerator [production]
14:46 <_joe_> also restarted pybal on lvs3003 [production]
14:42 <_joe_> restarted pybal on lvs1006 [production]
14:38 <hashar> restarting Jenkins [production]
14:32 <hashar> Jenkins lost a bunch of executors :/ [production]
14:30 <hashar> CI / Zuul stalled somehow [production]
12:56 <jynus> rolling restart, configuration upgrade of es1015 [production]
12:20 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Depool es1015; es1013 at 100% load; pool es1017 with low weight (duration: 00m 28s) [production]
11:08 <YuviPanda> restarting pdns on holmium [production]
10:52 <jynus> database and system maintenance to es1017 [production]
10:43 <hashar> CI / zuul / nodepool recovered. Root cause was some malfunction in openstack wmflabs [production]
10:20 <YuviPanda> restarted nova-conductor and scheduler on labcontrol1001 [production]
10:07 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Repool es1013 (lower weight for now) and depool es1017 (duration: 00m 41s) [production]
10:05 <hashar> stopped Nodepool. Can not create instances anymore on wmflabs ( https://phabricator.wikimedia.org/T120586 ) [production]
09:46 <hashar> restarting Nodepool on labnodepool1001.eqiad.wment [production]
09:40 <hashar> CI / Zuul stalled. Nodepool can no more spawn instances :-/ [production]
09:27 <godog> nodetool decommission restbase1008 [production]
09:13 <jynus> es1013 maintenance (mysql restart, upgrade, possible reboot) [production]
08:27 <_joe_> uploaded etcd 2.2 package from stretch to jessie-wikimedia [production]
03:56 <l10nupdate@tin> ResourceLoader cache refresh completed at Mon Dec 7 03:56:49 UTC 2015 (duration 1h 32m 22s) [production]
02:24 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.7) (duration: 09m 59s) [production]
2015-12-06 §
21:48 <ori> krypton unresponsive, nothing on console. shutting down, increasing instance ram from 2 to 4g, and rebooting. [production]
18:49 <legoktm> reset auth token for User:QuimGil [production]
05:50 <mutante> silver gzip /var/log/nutcracker.log.1 [production]
05:40 <mutante> silver: apt-get clean for disk space [production]
03:57 <l10nupdate@tin> ResourceLoader cache refresh completed at Sun Dec 6 03:57:02 UTC 2015 (duration 1h 31m 41s) [production]
02:25 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.7) (duration: 10m 04s) [production]
2015-12-05 §
18:30 <gwicke> started nodetool decommission on restbase1008 [production]
11:35 <reedy@tin> Synchronized wmf-config/CommonSettings.php: Disable common password password policy to come in wmf.8 (duration: 00m 28s) [production]
11:23 <reedy@tin> Purged l10n cache for 1.27.0-wmf.5 [production]
11:22 <reedy@tin> Synchronized php-1.27.0-wmf.7/extensions/WikimediaMaintenance/refreshMessageBlobs.php: Less waiting for slaves (duration: 00m 28s) [production]