|
2014-10-07
§
|
| 14:31 |
<reedy> |
Synchronized docroot and w: Fixup noc (duration: 00m 16s) |
[production] |
| 10:53 |
<godog> |
restart commons swiftrepl from ms-fe1003 and non-commons from ms-fe1004 to avoid maxing out copper's nic |
[production] |
| 10:09 |
<godog> |
start swiftrepl of commons originals eqiad -> codfw |
[production] |
| 09:56 |
<godog> |
start swiftrepl of non-commons originals eqiad -> codfw |
[production] |
| 08:55 |
<Krinkle> |
Building additional contint slaves in labs (integration-slave1004 with precise and integration-slave1009 with trusty) |
[production] |
| 08:21 |
<Krinkle> |
Reload Zuul to deploy 5e905e7c9dde9f47482d |
[production] |
| 06:02 |
<ori> |
Synchronized php-1.25wmf1/includes/objectcache/HashBagOStuff.php: I0b0b5f01: HashBagOStuff: use the value itself as the CAS token (duration: 00m 06s) |
[production] |
| 06:02 |
<ori> |
Synchronized php-1.25wmf2/includes/objectcache/HashBagOStuff.php: I0b0b5f01: HashBagOStuff: use the value itself as the CAS token (duration: 00m 07s) |
[production] |
| 03:28 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Tue Oct 7 03:28:09 UTC 2014 (duration 28m 8s) |
[production] |
| 02:26 |
<LocalisationUpdate> |
completed (1.25wmf2) at 2014-10-07 02:26:04+00:00 |
[production] |
| 02:14 |
<LocalisationUpdate> |
completed (1.25wmf1) at 2014-10-07 02:14:21+00:00 |
[production] |
| 01:16 |
<ori> |
Synchronized php-1.25wmf1/extensions/Wikidata: Ie92da71 / I44f1dce: Update Wikidata, fixes for serialization issues (duration: 00m 09s) |
[production] |
| 01:15 |
<ori> |
Synchronized php-1.25wmf2/extensions/Wikidata: Ie92da71 / I44f1dce: Update Wikidata, fixes for serialization issues (duration: 00m 10s) |
[production] |
| 00:34 |
<Tim> |
core dumps were enabled on mw1088, unexpectedly started gathering natural segfault traffic |
[production] |
|
2014-10-06
§
|
| 23:49 |
<mutante> |
tarin, nfs-1 - revoked salt key,puppet cert, stored configs |
[production] |
| 23:06 |
<maxsem> |
Synchronized php-1.25wmf2/extensions/MobileFrontend/: (no message) (duration: 00m 04s) |
[production] |
| 23:06 |
<maxsem> |
Synchronized php-1.25wmf1/extensions/MobileFrontend/: (no message) (duration: 00m 04s) |
[production] |
| 23:05 |
<maxsem> |
Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/165099/ https://gerrit.wikimedia.org/r/#/c/164902/ (duration: 00m 04s) |
[production] |
| 22:28 |
<Tim> |
on mw1088 debugging crash bug 71542 |
[production] |
| 22:00 |
<hoo> |
Reverted wd:Q17939676 to 157541810 and edit=sysop |
[production] |
| 21:47 |
<cscott> |
updated OCG to version bbdf4c6400cfbbc6030114ad16e1a6f7025eab2c |
[production] |
| 20:21 |
<awight> |
backfilled recurring GC glitch from FR #2018, 3342 records affected. |
[production] |
| 20:16 |
<subbu> |
deployed parsoid sha 13a53ab3 (deploy repo sha 38d44ada7) |
[production] |
| 19:04 |
<akosiaris> |
issued cf disable and halt on nas1-a.pmtpa.wmnet nas1-b.pmtpa.wmnet. They are officially down :) |
[production] |
| 17:22 |
<_joe_> |
hhvm load test finished |
[production] |
| 17:03 |
<_joe_> |
depooling and repooling progressively hhvm appservers to do see performance under load |
[production] |
| 16:57 |
<Krinkle> |
git-deploy: Deploying integration/slave-scripts 0b85d48 |
[production] |
| 16:19 |
<ori> |
Synchronized wmf-config/CommonSettings.php: Set $wgPercentHHVM to 1 (duration: 00m 27s) |
[production] |
| 15:54 |
<cscott> |
updated OCG to version aee3712b352f51f96569de0bcccf3facf654e688 |
[production] |
| 15:45 |
<GroggyPanda> |
deleted graphite data for deployment-rsync02 by hand on labmon1001, since instance has been dead. Need to move to shinken + dynamic host.cfg |
[production] |
| 15:37 |
<manybubbles> |
Synchronized php-1.25wmf1/extensions/Wikidata/: SWAT update wikidata (duration: 00m 10s) |
[production] |
| 15:23 |
<manybubbles> |
Synchronized php-1.25wmf2/extensions/Wikidata/: SWAT update wikidata (duration: 00m 10s) |
[production] |
| 15:22 |
<hashar> |
Zuul jobs proceeding again |
[production] |
| 15:22 |
<godog> |
swiftrepl replicating non-sharded originals containers eqiad -> codfw |
[production] |
| 15:22 |
<manybubbles> |
Synchronized wmf-config/: SWAT Add tracking categories for files with attribution problems (duration: 00m 06s) |
[production] |
| 15:19 |
<cscott> |
ran 'sudo -u ocg -g ocg nodejs-ocg scripts/run-garbage-collect.js -c /home/cscott/config.js' from /home/cscott/ocg/mw-ocg-service in order to clear caches (working around https://gerrit.wikimedia.org/r/164644 ) on ocg100x.eqiad.wmnet |
[production] |
| 14:51 |
<cmjohnson1> |
disconnecting Tampa servers |
[production] |
| 13:46 |
<godog> |
starting test swiftrepl run on wikibooks eqiad -> codfw |
[production] |
| 11:49 |
<_joe_> |
done restarting ocg servers |
[production] |
| 11:34 |
<_joe_> |
rolling restart and cleaning of ocg nodes, trying to unlock pdf generation |
[production] |
| 11:11 |
<mark> |
Shutdown tarin |
[production] |
| 11:11 |
<mark> |
Shutdown sanger |
[production] |
| 09:27 |
<_joe_> |
cleaned ocg another time |
[production] |
| 09:07 |
<mark> |
Stopped dovecot on sanger |
[production] |
| 08:06 |
<_joe_> |
cleaned ocg1001, again |
[production] |
| 05:57 |
<Nemo_bis> |
"book creator seems stuck": PDF servers at 97 % CPU, little traffic, enough disk free for about 1 day more |
[production] |
| 03:26 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Oct 6 03:26:31 UTC 2014 (duration 26m 30s) |
[production] |
| 02:59 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1060 (duration: 00m 06s) |
[production] |
| 02:28 |
<LocalisationUpdate> |
completed (1.25wmf2) at 2014-10-06 02:28:42+00:00 |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.25wmf1) at 2014-10-06 02:17:40+00:00 |
[production] |