|
2014-10-03
§
|
| 20:30 |
<cscott> |
(the above was on ocg100x.eqiad.wmnet) |
[production] |
| 20:30 |
<cscott> |
ran 'sudo -u ocg -g ocg nodejs-ocg scripts/run-garbage-collect.js -c /home/cscott/config.js' from /home/cscott/ocg/mw-ocg-service in order to clear caches (working around https://gerrit.wikimedia.org/r/164644 ) |
[production] |
| 20:23 |
<andrewbogott> |
restarted zuul |
[production] |
| 20:23 |
<ori> |
Synchronized wmf-config/mc.php: Ie1ed821a7: Set bloom cache config (duration: 00m 03s) |
[production] |
| 19:34 |
<ori> |
when running puppet merge: fatal: Unable to create '/var/lib/git/operations/puppet/.git/refs/remotes/origin/production.lock': File exists. |
[production] |
| 19:28 |
<hashar> |
Restarting Zuul sorry :-/ |
[production] |
| 19:26 |
<hashar> |
Zuul in some kind of death loop |
[production] |
| 19:05 |
<mutante> |
purging old 'searchqa' scripts and logs from iron (gerrit 164429 removes from puppet) |
[production] |
| 18:46 |
<mutante> |
restored dbtree from manual backup (should have been synced by scripts) |
[production] |
| 18:25 |
<aaron> |
Synchronized php-1.25wmf2/extensions/CentralAuth: (no message) (duration: 00m 05s) |
[production] |
| 18:14 |
<maxsem> |
Synchronized php-1.25wmf2/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/#/c/164543/ (duration: 00m 04s) |
[production] |
| 18:14 |
<maxsem> |
Synchronized php-1.25wmf1/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/#/c/164543/ (duration: 00m 04s) |
[production] |
| 17:53 |
<bblack> |
restarted parsoid varnishes |
[production] |
| 17:30 |
<andrewbogott> |
enabling puppet on tungsten which is disabled for mysterious reasons |
[production] |
| 17:04 |
<gwicke> |
restarting parsoids after CPU spike |
[production] |
| 16:50 |
<ejegg> |
updated crm from 6f66294607d132230ef82fb4867c37a8700bfd4e to ef68d9fe98a64e819ebbdddbe5e13f83037607ce |
[production] |
| 15:34 |
<_joe_> |
purging varnish cache for parsoid (RT 8528) |
[production] |
| 15:19 |
<mark> |
shutdown pdf2 & pdf3 |
[production] |
| 15:15 |
<bblack> |
adding 10.2.1.0/24 aggregate in cr-[12].codfw |
[production] |
| 15:14 |
<bblack> |
dropping 10.2.1.0/24 aggregate + static routes in cr2-pmtpa |
[production] |
| 14:37 |
<bblack> |
testing dns server upgrade on baham |
[production] |
| 14:35 |
<manybubbles> |
Synchronized wmf-config/CirrusSearch-labs.php: Noop - just keeps beta config in sync (duration: 00m 04s) |
[production] |
| 14:18 |
<Jeff_Green> |
launched iodine:/opt/otrs/bin/otrs.RebuildFulltextIndex.pl per bugzilla #64473 |
[production] |
| 13:07 |
<Reedy> |
Updated minor_mime to varbinary(100) on image|filearchive|oldimage on foundationwiki |
[production] |
| 12:31 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool es1004 (duration: 00m 06s) |
[production] |
| 11:51 |
<reedy> |
Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 15s) |
[production] |
| 11:49 |
<aude> |
Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 10s) |
[production] |
| 11:37 |
<springle> |
shutdown db60 db68 db69 db71 db72 db73 db74 es4 es7 es10 |
[production] |
| 10:43 |
<mark> |
Shutdown amaranth.toolserver.org's switchport on asw-d-pmtpa |
[production] |
| 09:35 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1063 (duration: 00m 07s) |
[production] |
| 08:31 |
<akosiaris> |
deleting snmp community from nas1-a, nas1-b. I guess librenms is going to start complaining |
[production] |
| 08:20 |
<akosiaris> |
unexporting, offline, destroying /vol/home_pmtpa on nas1-a |
[production] |
| 06:59 |
<_joe_> |
depooling mw1022, then reimaging it |
[production] |
| 04:22 |
<springle> |
upgrade db1063 mariadb 10 |
[production] |
| 04:16 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Oct 3 04:16:35 UTC 2014 (duration 16m 34s) |
[production] |
| 04:04 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1063 (duration: 00m 08s) |
[production] |
| 03:17 |
<LocalisationUpdate> |
completed (1.25wmf2) at 2014-10-03 03:17:28+00:00 |
[production] |
| 02:44 |
<LocalisationUpdate> |
completed (1.25wmf1) at 2014-10-03 02:44:39+00:00 |
[production] |
|
2014-10-02
§
|
| 23:15 |
<maxsem> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/164483 (duration: 00m 03s) |
[production] |
| 23:06 |
<maxsem> |
Synchronized php-1.25wmf2/extensions/MobileFrontend/: (no message) (duration: 00m 04s) |
[production] |
| 22:53 |
<reedy> |
Synchronized wmf-config/: Experimentally enable vips for larger (>50MP) tiff files (duration: 00m 15s) |
[production] |
| 22:38 |
<mutante> |
icinga_broken_due_to_missing_hostgroup_counter incremented |
[production] |
| 21:43 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: Disabling prerendering of images from this mornings swat (duration: 00m 04s) |
[production] |
| 21:33 |
<bblack> |
added LVS BGP config setup to cr[12]-codfw |
[production] |
| 20:46 |
<ori> |
Synchronized php-1.24wmf22/extensions/NavigationTiming: Update NavigationTiming for cherry-picks (duration: 00m 03s) |
[production] |
| 20:44 |
<ori> |
Synchronized php-1.25wmf1/extensions/NavigationTiming: Update NavigationTiming for cherry-picks (duration: 00m 04s) |
[production] |
| 20:11 |
<bblack> |
stopping -> starting uwsgi/apache -type stuff on tungsten |
[production] |
| 19:55 |
<aude> |
populated sites table for fawikivoyage |
[production] |
| 19:27 |
<bd808> |
hosts that failed Trebuchet update of scap: virt0.wikimedia.org, fenari.wikimedia.org, mw1110.eqiad.wmnet, mw1053.eqiad.wmnet. mw1053.eqiad.wmnet only failed checkout |
[production] |
| 19:23 |
<bd808> |
Trebuchet reports for scap sync "231/234 minions completed fetch; 230/234 minions completed checkout" Some stale entries need to be removed from Trebuchet redis cache |
[production] |