3301-3350 of 10000 results (30ms)
2015-12-07 §
16:31 <_joe_> upgrading pybal on all the backup LVSs in codfw [production]
16:28 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: Rights configuration on fa.wikipedia [[gerrit:254486]] (duration: 00m 29s) [production]
16:28 <_joe_> upgrading pybal on lvs1007-12 [production]
16:19 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration for ur.wikipedia [[gerrit:254665]] (duration: 00m 30s) [production]
16:18 <_joe_> uploaded pybal 1.13.2 to reprepro [production]
16:14 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: Translate project namesapce for jbowiki [[gerrit:255954]] (duration: 00m 28s) [production]
15:39 <moritzm> uploaded openssl 1.0.2e-1~wmf1 to carbon [production]
15:37 <yurik> deployed kartotherian [production]
15:25 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Depool es1019; es1017 at 100% load; pool es1015 with low weight (duration: 00m 28s) [production]
14:56 <yurik> deployed latest tilerator [production]
14:46 <_joe_> also restarted pybal on lvs3003 [production]
14:42 <_joe_> restarted pybal on lvs1006 [production]
14:38 <hashar> restarting Jenkins [production]
14:32 <hashar> Jenkins lost a bunch of executors :/ [production]
14:30 <hashar> CI / Zuul stalled somehow [production]
12:56 <jynus> rolling restart, configuration upgrade of es1015 [production]
12:20 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Depool es1015; es1013 at 100% load; pool es1017 with low weight (duration: 00m 28s) [production]
11:08 <YuviPanda> restarting pdns on holmium [production]
10:52 <jynus> database and system maintenance to es1017 [production]
10:43 <hashar> CI / zuul / nodepool recovered. Root cause was some malfunction in openstack wmflabs [production]
10:20 <YuviPanda> restarted nova-conductor and scheduler on labcontrol1001 [production]
10:07 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Repool es1013 (lower weight for now) and depool es1017 (duration: 00m 41s) [production]
10:05 <hashar> stopped Nodepool. Can not create instances anymore on wmflabs ( https://phabricator.wikimedia.org/T120586 ) [production]
09:46 <hashar> restarting Nodepool on labnodepool1001.eqiad.wment [production]
09:40 <hashar> CI / Zuul stalled. Nodepool can no more spawn instances :-/ [production]
09:27 <godog> nodetool decommission restbase1008 [production]
09:13 <jynus> es1013 maintenance (mysql restart, upgrade, possible reboot) [production]
08:27 <_joe_> uploaded etcd 2.2 package from stretch to jessie-wikimedia [production]
03:56 <l10nupdate@tin> ResourceLoader cache refresh completed at Mon Dec 7 03:56:49 UTC 2015 (duration 1h 32m 22s) [production]
02:24 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.7) (duration: 09m 59s) [production]
2015-12-06 §
21:48 <ori> krypton unresponsive, nothing on console. shutting down, increasing instance ram from 2 to 4g, and rebooting. [production]
18:49 <legoktm> reset auth token for User:QuimGil [production]
05:50 <mutante> silver gzip /var/log/nutcracker.log.1 [production]
05:40 <mutante> silver: apt-get clean for disk space [production]
03:57 <l10nupdate@tin> ResourceLoader cache refresh completed at Sun Dec 6 03:57:02 UTC 2015 (duration 1h 31m 41s) [production]
02:25 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.7) (duration: 10m 04s) [production]
2015-12-05 §
18:30 <gwicke> started nodetool decommission on restbase1008 [production]
11:35 <reedy@tin> Synchronized wmf-config/CommonSettings.php: Disable common password password policy to come in wmf.8 (duration: 00m 28s) [production]
11:23 <reedy@tin> Purged l10n cache for 1.27.0-wmf.5 [production]
11:22 <reedy@tin> Synchronized php-1.27.0-wmf.7/extensions/WikimediaMaintenance/refreshMessageBlobs.php: Less waiting for slaves (duration: 00m 28s) [production]
11:13 <reedy@tin> Synchronized docroot and w: Add jobqueue-labs to noc (duration: 00m 28s) [production]
08:59 <bblack> offlined db1019 megacli disk 32:11 [production]
06:09 <l10nupdate@tin> ResourceLoader cache refresh completed at Sat Dec 5 06:09:07 UTC 2015 (duration 3h 44m 18s) [production]
02:24 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.7) (duration: 09m 59s) [production]
2015-12-04 §
21:44 <andrewbogott> disabling puppet on labcontrol1002 for ldap testing [production]
21:36 <ori@tin> Synchronized php-1.27.0-wmf.7/includes/Hooks.php: Iba0138a: Don't install a custom error handler for hooks (T117553) (duration: 00m 28s) [production]
20:28 <ori@tin> Synchronized wmf-config/jobqueue-eqiad.php: Idee6a1980: job queue: use instances on port 6378 as aggregators (duration: 00m 30s) [production]
19:21 <ori> krypton: updated Grafana to 2.6.0-beta1 for bug fix for issue 3422 [production]
15:52 <Jeff_Green> add mx record for donate.wikimedia.org [production]
15:33 <godog> ms-be2019 rebooted by itself, ilo event log shows "Uncorrectable Machine Check Exception (Board 0, Processor 2, APIC ID 0x00000038, Bank 0x00000003, Status 0xFE000040'00020135, Address 0x00000000'FEB82F63, Misc 0x00000000'00002285)" [production]