|
2016-09-23
§
|
| 20:49 |
<halfak> |
restarted ores-uwsgi and precached services on ores-web-03 |
[ores] |
| 20:41 |
<yuvipanda> |
add halfak as projectadmin |
[quarry] |
| 20:21 |
<hashar> |
integration: salt -v '*trusty*' cmd.run 'service mysql start' |
[releng] |
| 20:00 |
<hashar> |
rebooting all CI permanent slaves. Making sure nothing is left on /mnt (which is no more mounted) |
[releng] |
| 19:53 |
<hashar> |
added a 30 minutes build timeout to https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/ |
[releng] |
| 18:41 |
<gehel> |
killing stuck tilerator notification processes on maps1001 - T145534 |
[production] |
| 17:57 |
<mutante> |
mira restarted cron |
[production] |
| 17:53 |
<ejegg> |
updated SmashPig from 8ac116037440746eaf64b9e99e1ee962d5d33475 to 372cd4008fee3fd02ad2eae9163cf7b28d2ef7c8 |
[production] |
| 17:46 |
<thcipriani@tin> |
Synchronized README: Test sync for new mira (duration: 01m 27s) |
[production] |
| 17:43 |
<mutante> |
mira - changing UID of l10nupdate to 10002, chown'ing files (1001 -> 10002) |
[production] |
| 17:35 |
<ebernhardson@tin> |
Synchronized php-1.28.0-wmf.20/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: Add timing marks to narrow down autocomplete timing regression (duration: 00m 50s) |
[production] |
| 17:31 |
<ebernhardson@tin> |
Synchronized php-1.28.0-wmf.20/extensions/CirrusSearch/includes/CompletionSuggester.php: Add timing marks to narrow down autocomplete timing regression (duration: 18m 43s) |
[production] |
| 17:04 |
<mutante> |
stat1002 - before it was hanging and then fixed due to https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hadoop/Administration#Fixing_HDFS_mount_at_.2Fmnt.2Fhdfs |
[production] |
| 17:03 |
<mutante> |
stat1002 - starting nagios-nrpe-server |
[production] |
| 15:02 |
<hashar> |
rebooting integration-slave-jessie-1001 |
[releng] |
| 14:55 |
<jynus> |
deployed dns update (removing db1010) T129395 |
[production] |
| 14:04 |
<hashar> |
remove the /mnt based tmpfs for T146381 / https://gerrit.wikimedia.org/r/#/c/312518/ via: salt -v '*' cmd.run 'umount /mnt/home/jenkins-deploy/tmpfs' |
[releng] |
| 13:41 |
<hashar> |
Switching tmpfs from /mnt to /srv https://gerrit.wikimedia.org/r/#/c/312330/ and running fab deploy_slave_scripts |
[releng] |
| 12:20 |
<moritzm> |
rearmed keyholder on mira |
[production] |
| 12:03 |
<_joe_> |
rolling restart of mw1280-90, high cpu usage due to memory leaks. |
[production] |
| 10:16 |
<moritzm> |
reimaging mira to jessie (again, previously installer config still pointed to trusty) |
[production] |
| 10:05 |
<Amir1> |
ladsgroup@terbium:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=wikidatawiki (T146461) and for 'trwiki', 'plwiki', 'fawiki', 'nlwiki', 'ruwiki', 'ptwiki' |
[production] |
| 10:00 |
<Amir1> |
ladsgroup@terbium:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=enwiki |
[production] |
| 09:58 |
<hashar@tin> |
Synchronized php-1.28.0-wmf.20/extensions/ORES/includes/Cache.php: No int typehinting (causes jobs to crash) T146461 (duration: 00m 42s) |
[production] |
| 09:58 |
<moritzm> |
rearmed keyholder on mira |
[production] |
| 09:48 |
<jynus> |
disabling alerts and shutting down db1010 in preparation for decommissioning T129395 |
[production] |
| 09:08 |
<moritzm> |
reimaging mira to jessie |
[production] |
| 09:06 |
<elukey> |
reboot eventlog2001.codfw.wmnet for kernel upgrades |
[analytics] |
| 09:06 |
<elukey> |
reboot eventlog2001.codfw.wmnet for kernel upgrades |
[production] |
| 08:52 |
<elukey> |
upgrading varnishkafka to 1.0.12-1 in cache:misc |
[production] |
| 08:45 |
<elukey> |
upgrading varnishkafka to 1.0.12-1 in cache:misc |
[analytics] |
| 08:44 |
<ema> |
depooled nginx restart on cp4003 and cp1045 for libssl upgrade |
[production] |
| 08:32 |
<elukey> |
upgrading varnishkafka to 1.0.12-1 in cache:maps |
[analytics] |
| 08:30 |
<elukey> |
upgrading varnishkafka to 1.0.12-1 in cache:maps |
[production] |
| 07:33 |
<elukey> |
executed 'find /var/log/hhvm/ -type f -user root -exec chown www-data:www-data {} \;' for all the api and appservers to remove/prevent cronspam (root:adm files also related to new reimaged hosts, Rsyslog needs to be configured before hhvm) - T132324 |
[production] |
| 07:02 |
<moritzm> |
rebooting francium for kernel security update |
[production] |
| 04:03 |
<aaron@tin> |
Synchronized php-1.28.0-wmf.20/includes/deferred: 5af1b93db1bb3d14844c55e4e3ed17fe963de551 (duration: 00m 48s) |
[production] |
| 04:02 |
<aaron@tin> |
Synchronized php-1.28.0-wmf.20/includes/libs/rdbms: 5af1b93db1bb3d14844c55e4e3ed17fe963de551 (duration: 00m 51s) |
[production] |
| 02:46 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Fri Sep 23 02:46:04 UTC 2016 (duration 6m 10s) |
[production] |
| 02:39 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.20) (duration: 17m 04s) |
[production] |
| 02:13 |
<maxsem@tin> |
Synchronized php-1.28.0-wmf.20/extensions/SecurePoll/: https://gerrit.wikimedia.org/r/#/c/312450/1 (duration: 00m 51s) |
[production] |
| 02:10 |
<mutante> |
mw1206, mw1224 - restarted hhvm and apache |
[production] |
| 01:49 |
<bblack> |
depooled mw1224 service apache2 |
[production] |
| 00:38 |
<Krenair> |
mw1224 apache stuck, not restarting for now in case someone wants to investigate later. possibly T89912? |
[production] |
| 00:17 |
<krenair@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/312339 (duration: 00m 48s) |
[production] |
| 00:16 |
<krenair@tin> |
Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/312339 (duration: 00m 47s) |
[production] |
| 00:08 |
<krenair@tin> |
Synchronized php-1.28.0-wmf.20/extensions/FlaggedRevs/business/RevisionReviewForm.php: https://gerrit.wikimedia.org/r/#/c/312423/ (duration: 00m 48s) |
[production] |