2201-2250 of 10000 results (25ms)
2016-06-23 §
12:07 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Depool db1059; Repool db1061 & db1062; increase weight of db1068 (duration: 00m 39s) [production]
11:32 <gehel> rolling restart of elasticsearch10(01|30|08|36|13|40) to activate new masters [production]
10:13 <andrewbogott> restarting rabbitmq-server on labcontrol1001 (random debugging attempt for T138106) [production]
09:49 <godog> reimage ms-be202[567] with incorrect raid settings [production]
09:11 <jynus> syncing etherpadlite.store (m1) on db2010, which had 2 bad chunks [production]
08:39 <mobrovac> change-prop restarting on scb to pick up ores rules https://gerrit.wikimedia.org/r/295576 [production]
08:06 <mobrovac> change-prop deploying 45db4f84827 [production]
06:59 <moritzm> installing spice security updates [production]
02:47 <l10nupdate@tin> ResourceLoader cache refresh completed at Thu Jun 23 02:47:59 UTC 2016 (duration 6m 44s) [production]
02:41 <mwdeploy@tin> scap sync-l10n completed (1.28.0-wmf.7) (duration: 07m 05s) [production]
02:26 <mwdeploy@tin> scap sync-l10n completed (1.28.0-wmf.6) (duration: 11m 19s) [production]
2016-06-22 §
23:24 <maxsem@tin> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/295560/ (duration: 00m 25s) [production]
23:23 <maxsem@tin> Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/295560/ (duration: 00m 24s) [production]
23:23 <maxsem@tin> Synchronized dblists/mobilemainpagelegacy.dblist: https://gerrit.wikimedia.org/r/#/c/295560/ (duration: 00m 24s) [production]
23:14 <maxsem@tin> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/294247/ (duration: 00m 24s) [production]
23:09 <maxsem@tin> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/295558/ (duration: 00m 40s) [production]
22:25 <ori> Ran hacked maintain-replicas.pl on labsdb100[13] for T135029 [production]
21:06 <bblack> cache perf: start deploy of -autocorking (probably last experiment I can squeeze in today) [production]
20:59 <Dereckson> Run namespaceDupes.php on ptwikinews (T138230) and frwikinews (T138442) [production]
20:33 <mdholloway> mobileapps: finished deploying 8046ee2 [production]
20:26 <yurik> deployed & restarted tilerator https://gerrit.wikimedia.org/r/#/c/295447/ [production]
20:25 <mdholloway> starting mobileapps deployment [production]
20:20 <Reedy> created tmplog_begin_devices on tmplog_end_devices on testwiki.cn_template_log [production]
20:18 <yurik> deployed & restarted kartotherian https://gerrit.wikimedia.org/r/#/c/295449/ [production]
19:32 <bblack> start rollout of first batch of cache sysctl stuff (un-mysterious + disable prequeue timestamps) [production]
19:29 <jynus> archiving and dropping reviewdb on m1 shard [production]
19:06 <thcipriani@tin> rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.7 [production]
18:46 <jynus> shutting down and reimaging db1001 [production]
18:20 <papaul> ms-be202[3-7] - signing puppet certs, salt-key, initial run [production]
17:23 <akosiaris> restart apache on ununpentium for m1 migration. Hosts RT, just did it for good measure [production]
17:21 <akosiaris> restarted bacula-director on helium [production]
17:15 <jynus> killing puppet, rt, librenms user connections on db1001 [production]
17:10 <jynus> failovered m1-master from db1001 to db1016 [production]
16:20 <gehel> new elasticsearch servers elastic1032-1047 are configured and have joined the eqiad cluster [production]
15:26 <thcipriani@tin> Synchronized php-1.28.0-wmf.6/extensions/OATHAuth: SWAT: [[gerrit:295510|Fixup qrcode-generating js, to stop race condition.]] (duration: 00m 33s) [production]
15:23 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:295514|Improve style]] (duration: 00m 33s) [production]
15:18 <thcipriani@tin> Synchronized php-1.28.0-wmf.7/extensions/OATHAuth: SWAT: [[gerrit:295511|Fixup qrcode-generating js, to stop race condition.]] (duration: 00m 27s) [production]
15:13 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:295478|Add www.wpc.ncep.noaa.gov to wgCopyUploadsDomains]] (duration: 00m 54s) [production]
15:01 <elukey> rebooting bohrium.eqiad.wmnet (running piwik) for kernel upgrades [production]
14:32 <jynus> checksumming m1 databases in preparation for failover [production]
14:29 <tgr> running https://phabricator.wikimedia.org/diffusion/ECAU/browse/master/maintenance/checkLocalUser.php for some users T119736 [production]
14:04 <moritzm> rolling restart of hhvm/apache on app servers in eqiad for expat security update [production]
13:42 <godog> add 500G to fluorine /a (almost full) [production]
13:31 <gehel> configuring new elasticsearch servers elastic1038-1042 in eqiad [production]
13:03 <hashar> Manually moved some missing build records. Restarting Jenkins [production]
12:49 <hashar> T80385 Restarting Jenkins with builds dir set to "${JENKINS_HOME}/builds/${ITEM_FULL_NAME}" which is /var/lib/jenkins/builds/XXX [production]
12:35 <gehel> starting reimage of mw1292 [production]
12:34 <_joe_> disabling puppet on mw1017, live-hacking it [production]
12:34 <hashar> T80385 stopping Jenkins and migrating all build records to /var/lib/jenkins/builds [production]
12:06 <gehel> configuring new elasticsearch servers elastic1033-1037 in eqiad [production]