|
2014-03-20
§
|
| 16:05 |
<bd808> |
Ran /usr/local/bin/sync-common && /usr/local/bin/scap-rebuild-cdbs on mw1163. Should not repool until it's back in the dsh group. Should me manually synced just before repooling. |
[production] |
| 15:36 |
<reedy> |
Finished scap: testwiki to 1.23wmf19 and build l10n cache (duration: 15m 32s) |
[production] |
| 15:21 |
<reedy> |
Started scap: testwiki to 1.23wmf19 and build l10n cache |
[production] |
| 15:03 |
<maxsem> |
synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/118328' |
[production] |
| 15:00 |
<bd808> |
logstash stopped ingesting logs at 2014-03-19T22:37:54.000Z. |
[production] |
| 14:57 |
<bd808> |
restarted logstash on logstash1001.eqiad.wmnet |
[production] |
| 03:23 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Mar 20 03:23:37 UTC 2014 (duration 23m 35s) |
[production] |
| 02:32 |
<LocalisationUpdate> |
completed (1.23wmf18) at 2014-03-20 02:32:10+00:00 |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.23wmf17) at 2014-03-20 02:17:34+00:00 |
[production] |
| 01:26 |
<Tim> |
gave Sam Reed access to racktables |
[production] |
| 00:26 |
<bd808> |
scap-slave rsync servers have "hosts allow = 10.0.0.0/16 10.64.0.0/22 10.64.16.0/22 10.64.32.0/22 208.80.152.0/22"; missing new 10.64.48.0/22 subnet |
[production] |
| 00:21 |
<bd808> |
Subnet for row D still not in rsync server config? |
[production] |
| 00:21 |
<bd808> |
scaps failed because "@ERROR: access denied to common from mw1202.eqiad.wmnet (10.64.48.34)" |
[production] |
| 00:16 |
<bd808> |
finished dsh -c -M -m mw1201,mw1202,mw1203,mw1208,mw1209,mw1210 -- '/usr/local/bin/sync-common; /usr/local/bin/scap-rebuild-cdbs' |
[production] |
| 00:13 |
<bd808> |
dsh -c -M -m mw1201,mw1202,mw1203,mw1208,mw1209,mw1210 -- '/usr/local/bin/sync-common; /usr/local/bin/scap-rebuild-cdbs' |
[production] |
|
2014-03-19
§
|
| 23:57 |
<spage> |
synchronized wmf-config 'Config change to enable extension Popups (Hovercards) on mediawikiwiki' |
[production] |
| 23:30 |
<spage> |
Finished scap: Retry Add Popups extension (Hovercards Beta Feature) to wmf17 and 18, not enabled yet (duration: 07m 15s) |
[production] |
| 23:22 |
<spage> |
Started scap: Retry Add Popups extension (Hovercards Beta Feature) to wmf17 and 18, not enabled yet |
[production] |
| 23:22 |
<spage> |
scap aborted: Add Popups extension (Hovercards Beta Feature) to wmf17 and 18, not enabled yet (duration: 69m 26s) |
[production] |
| 23:20 |
<bd808> |
restarted elasticsearch on logstash1002 |
[production] |
| 23:18 |
<bd808> |
scap by spage looks hung with 50 hosts not finished rsyncing |
[production] |
| 22:38 |
<bd808> |
Manually ran sync-common and scap-rebuild-cdbs on mw1163 |
[production] |
| 22:33 |
<mutante> |
depooling mw1163 |
[production] |
| 22:28 |
<ori> |
synchronized wmf-config/mc.php 'Set serializer to php for production memcache' |
[production] |
| 22:12 |
<spage> |
Started scap: Add Popups extension (Hovercards Beta Feature) to wmf17 and 18, not enabled yet |
[production] |
| 22:11 |
<spage> |
updated /a/common to {{Gerrit|I3d59b8246}}: Add new Popups extension to list |
[production] |
| 20:45 |
<subbu> |
deployed Parsoid ff8c49e9 with deploy 20923176 (a better log message then the previous one logged at 20:40) |
[production] |
| 20:40 |
<subbu> |
Deployed parsoid/deploy 20923176 |
[production] |
| 20:39 |
<robh> |
parsoid service successfully restarted across hosts via dsh |
[production] |
| 20:06 |
<yurik> |
synchronized docroot/bits/WikipediaMobileFirefoxOS/ |
[production] |
| 20:03 |
<mutante> |
shut down sq67-70 |
[production] |
| 20:00 |
<akosiaris> |
disabled puppet on labsdb1004 until osm2pgsql completes |
[production] |
| 19:21 |
<mutante> |
sq67-70: disable puppet,revoke puppet certs,delete salt keys and stored configs (delete from icinga) |
[production] |
| 19:15 |
<mutante> |
shutting down ssl1-4 |
[production] |
| 18:59 |
<bblack> |
all varnishes upgraded to 3.0.5plus-wmftest-wm5 (without restart, just adds header vmod) |
[production] |
| 18:46 |
<mutante> |
ssl1-4: puppet agent --disable, puppetstoredconfigclean, revoking puppet certs |
[production] |
| 18:24 |
<cmjohnson1> |
powering down mw1183 to reseat dimm |
[production] |
| 17:56 |
<cmjohnson1> |
mw1177 powering down to reseat DIMM |
[production] |
| 17:42 |
<awight> |
tools updated from 87dbe60d2557a38553ae2bb602f777b4096915b8 to de586ae4ad03be52da1bb24177ea4bf7cdbd2c94 |
[production] |
| 17:32 |
<mutante> |
DNS update - removing Tampa appservers |
[production] |
| 17:15 |
<awight> |
updated crm from f8b2dab4689ae096898a21f9b3b57d505aa23f5e to 8c8f0de23baa800c2d3ad47056bdb011b1beb9d3 |
[production] |
| 13:51 |
<hashar> |
Jenkins : applying crazy node/npm upgrade hack on Jenkins labs instances integration-slave1001 and integration-slave1002 ( ref: https://bugzilla.wikimedia.org/show_bug.cgi?id=61508#c2 and ops list) |
[production] |
| 12:39 |
<manybubbles> |
reindexed all group0 wikis now that we have the Cirrus SWAT deploy (from yesterday). Once the reindex is done we can deploy code to improve performance. Yummy. Starting on commons and eventually doing the rest of group1. |
[production] |
| 11:24 |
<hashar> |
Jenkins: all npm based jobs are broken due to nodejs self signed certificate being outdated on contint labs instances (see {{bug|61508}} and mail to ops list) |
[production] |
| 10:43 |
<hashar> |
Jenkins pooling integration-slave1002.eqiad.wmflabs |
[production] |
| 10:36 |
<hashar> |
Jenkins attempting to bring up integration-slave1001.eqiad.wmflabs (aka migrating Jenkins slave nodes in labs from pmtpa to eqiad) |
[production] |
| 09:13 |
<ori> |
synchronized wmf-config/mc-labs.php 'I46a9d180b: Beta cluster MemcachedPeclBagOStuff: use PHP serialization' |
[production] |
| 08:54 |
<springle> |
synchronized wmf-config/db-eqiad.php 's4 repool db1042' |
[production] |
| 02:43 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Mar 19 02:43:28 UTC 2014 (duration 43m 27s) |
[production] |
| 02:12 |
<LocalisationUpdate> |
completed (1.23wmf18) at 2014-03-19 02:12:42+00:00 |
[production] |