151-200 of 10000 results (20ms)
2017-03-29 §
13:48 <elukey> upgrading ssl cert rendering.svc.eqiad.wmnet to include the new discovery endpoints [production]
13:08 <reedy@tin> Synchronized wmf-config/CommonSettings.php: use wfLoadExtension for VisualEditor (duration: 00m 44s) [production]
12:53 <elukey> reimage analytics1045 to Debian Jessie [production]
12:52 <_joe_> depooling wtp1001 to test puppet/confd transfer of responsibilities [production]
11:47 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1094 - T17441 (duration: 00m 44s) [production]
11:30 <hoo> Started a Wikidata JSON dump run on snapshot1007 using Zend (due to T161695). [production]
11:16 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1094 - T17441 (duration: 00m 44s) [production]
11:03 <elukey> upgrading ssl cert appservers.svc.codfw.wmnet to include the new discovery endpoints [production]
11:01 <moritzm> Linux 4.9 uploaded for jessie-wikimedia (along with new meta package linux-meta-4.9 and updated firmware) [production]
11:01 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1093 - T17441 (duration: 00m 44s) [production]
10:27 <godog> reimage netmon1001 with jessie [production]
10:12 <ema> emptying /srv/log/parsoid/main.log.1 (3.2G!) on ruthenium to reclaim some disk space [production]
10:11 <elukey> upgrading ssl cert api.svc.codfw.wmnet to include the new discovery endpoints [production]
09:39 <ema> upgrading twisted to 16.2.0 on lvs200[123] (codfw primaries) T160433 [production]
08:54 <ema> upgrading twisted to 16.2.0 on lvs200[456] (codfw secondaries) T160433 [production]
08:39 <ema> apt.w.o: set digest-algo to sha256 in gpg.conf T132325 [production]
08:29 <elukey> upgrading ssl cert rendering.svc.codfw.wmnet to include the new discovery endpoints [production]
07:57 <marostegui> Convert s6 UNIQUE keys into PK on db1093 - T17441 [production]
07:55 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1093 - T17441 (duration: 00m 54s) [production]
06:01 <marostegui> Keep converting UNIQUE keys to PK on s4 - db1091 - T17441 [production]
03:15 <l10nupdate@tin> ResourceLoader cache refresh completed at Wed Mar 29 03:15:16 UTC 2017 (duration 5m 53s) [production]
03:09 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.18) (duration: 14m 55s) [production]
02:35 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.17) (duration: 13m 41s) [production]
01:41 <krinkle@tin> Synchronized errorpages/404.php: Match 404.html and default.html - Id58e25afbe (duration: 00m 44s) [production]
01:16 <mutante> rsyncing librenms/torrus/smokeping app data from netmon1001 to gerrit2001. adding alias "syncit" to do it all at once (T125020) [production]
00:57 <paravoid> Removing upload.wikimedia.org/index.html ("swift delete root index.html") from both eqiad/codfw [production]
2017-03-28 §
23:22 <thcipriani@tin> Synchronized php-1.29.0-wmf.17/extensions/NavigationTiming/modules/ext.navigationTiming.js: SWAT: [[gerrit:345078|ext.NavigationTiming: Restore unsampled Save Timing]] T161368 (duration: 00m 45s) [production]
23:16 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:345184|Enable header version 2 on all wikis]] T160471 (duration: 00m 45s) [production]
22:45 <urandom> T111113: Restarting Cassandra instances, eqiad row 'd' {{done]} [production]
22:21 <mutante> DNS - creating new language "dty" (T161529) - running "authdns-gen-zones -f /srv/authdns/git/templates /etc/gdnsd/zones && gdnsd checkconf && gdnsd reload-zones" to trigger re-creation of zone files after change in langs.tmpl. (gerrit:345077) | https://www.ethnologue.com/language/dty [production]
22:19 <mutante> DNS - creating new language "dty" (T160865) - running "authdns-gen-zones -f /srv/authdns/git/templates /etc/gdnsd/zones && gdnsd checkconf && gdnsd reload-zones" to trigger re-creation of zone files after change in langs.tmpl. (gerrit:345077) | https://www.ethnologue.com/language/dty [production]
21:55 <urandom> T111113: Restarting Cassandra instances, eqiad row 'd' [production]
21:55 <urandom> T111113: Restarting Cassandra instances, eqiad row 'b' {{done}} [production]
21:18 <andrewbogott> upgraded nova-compute on labvirt1014 because it contains a long-awaited bugfix [production]
21:08 <urandom> T111113: Restarting Cassandra instances, eqiad row 'b' [production]
21:08 <urandom> T111113: Restarting Cassandra instances, eqiad row 'a' {{done}} [production]
20:24 <mutante> ms-fe1001 thru msfe1004 - scheduled last downtime for host and services in icinga - shutdown -h now, turn them off, revoke puppet certs, salt-keys... (T160986) [production]
20:22 <mutante> mc1019 - puppet fail due to Failed resource /etc/redis/replica since 4 days [production]
20:21 <urandom> T111113: Restarting Cassandra instances, eqiad row 'a' [production]
20:21 <mutante> copper - puppet errors due to Failed resource /var/lib/docker/devicemapper ?? [production]
20:19 <mutante> mwdebug1002 - same, was low on disk space, 'apt-get clean' freed > 3GB [production]
20:18 <mutante> mwdebug1001 - was low on disk space, 'apt-get clean' - freed about 4GB [production]
20:15 <mutante> mw1261 - depooled [production]
20:14 <mutante> mw1261 runs with HHVM 3.18 - which seems to have a bug leading to a deadlock every 4-5 hours [production]
20:14 <thcipriani@tin> rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.29.0-wmf.18 [production]
20:13 <mutante> mw1261 HHVM crash as predicted by Moritz - ran sudo hhvm-dump-debug. Backtrace saved as /tmp/hhvm.79460.bt. [production]
20:06 <mutante> ms-fe100[1-4] - disable/stop puppet, stop salt minion, decom (T160986) [production]
19:57 <thcipriani@tin> Finished scap: testwiki to php-1.29.0-wmf.18 and rebuild l10n cache (duration: 40m 19s) [production]
19:37 <mobrovac> restbase deploying d477f495 [production]
19:33 <urandom> T111113: Restarting Cassandra instances, codfw row 'd' {{done}} [production]