|
2014-03-20
§
|
| 20:08 |
<mutante> |
DNS update - remove sq67-70, former varnish testing |
[production] |
| 19:35 |
<akosiaris> |
created a 50G LV for /var/log on zirconium, stopped all services, moved data to it, mounted it and restarted all services |
[production] |
| 19:29 |
<reedy> |
synchronized wmf-config/ 'Wikibase config updates' |
[production] |
| 19:23 |
<reedy> |
Finished scap: Rebuild 1.23wmf19 l10n cache for wikibase (duration: 12m 01s) |
[production] |
| 19:10 |
<reedy> |
Started scap: Rebuild 1.23wmf19 l10n cache for wikibase |
[production] |
| 19:08 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.23wmf19 |
[production] |
| 19:02 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.23wmf18 |
[production] |
| 19:00 |
<mutante> |
disk full on zirconium - gzipping some etherpadlite.sql dump i found |
[production] |
| 17:41 |
<reedy> |
synchronized php-1.23wmf19 'Update Wikidata and WikimediaMessages' |
[production] |
| 17:06 |
<Krinkle> |
Reloading Zuul to deploy Ie800ed90b51c47d5a1 |
[production] |
| 16:58 |
<mutante> |
repooling mw1163 (it's back in dsh as well) |
[production] |
| 16:38 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.23wmf18 till window |
[production] |
| 16:05 |
<bd808> |
Ran /usr/local/bin/sync-common && /usr/local/bin/scap-rebuild-cdbs on mw1163. Should not repool until it's back in the dsh group. Should me manually synced just before repooling. |
[production] |
| 15:36 |
<reedy> |
Finished scap: testwiki to 1.23wmf19 and build l10n cache (duration: 15m 32s) |
[production] |
| 15:21 |
<reedy> |
Started scap: testwiki to 1.23wmf19 and build l10n cache |
[production] |
| 15:03 |
<maxsem> |
synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/118328' |
[production] |
| 15:00 |
<bd808> |
logstash stopped ingesting logs at 2014-03-19T22:37:54.000Z. |
[production] |
| 14:57 |
<bd808> |
restarted logstash on logstash1001.eqiad.wmnet |
[production] |
| 03:23 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Mar 20 03:23:37 UTC 2014 (duration 23m 35s) |
[production] |
| 02:32 |
<LocalisationUpdate> |
completed (1.23wmf18) at 2014-03-20 02:32:10+00:00 |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.23wmf17) at 2014-03-20 02:17:34+00:00 |
[production] |
| 01:26 |
<Tim> |
gave Sam Reed access to racktables |
[production] |
| 00:26 |
<bd808> |
scap-slave rsync servers have "hosts allow = 10.0.0.0/16 10.64.0.0/22 10.64.16.0/22 10.64.32.0/22 208.80.152.0/22"; missing new 10.64.48.0/22 subnet |
[production] |
| 00:21 |
<bd808> |
Subnet for row D still not in rsync server config? |
[production] |
| 00:21 |
<bd808> |
scaps failed because "@ERROR: access denied to common from mw1202.eqiad.wmnet (10.64.48.34)" |
[production] |
| 00:16 |
<bd808> |
finished dsh -c -M -m mw1201,mw1202,mw1203,mw1208,mw1209,mw1210 -- '/usr/local/bin/sync-common; /usr/local/bin/scap-rebuild-cdbs' |
[production] |
| 00:13 |
<bd808> |
dsh -c -M -m mw1201,mw1202,mw1203,mw1208,mw1209,mw1210 -- '/usr/local/bin/sync-common; /usr/local/bin/scap-rebuild-cdbs' |
[production] |
|
2014-03-19
§
|
| 23:57 |
<spage> |
synchronized wmf-config 'Config change to enable extension Popups (Hovercards) on mediawikiwiki' |
[production] |
| 23:30 |
<spage> |
Finished scap: Retry Add Popups extension (Hovercards Beta Feature) to wmf17 and 18, not enabled yet (duration: 07m 15s) |
[production] |
| 23:22 |
<spage> |
Started scap: Retry Add Popups extension (Hovercards Beta Feature) to wmf17 and 18, not enabled yet |
[production] |
| 23:22 |
<spage> |
scap aborted: Add Popups extension (Hovercards Beta Feature) to wmf17 and 18, not enabled yet (duration: 69m 26s) |
[production] |
| 23:20 |
<bd808> |
restarted elasticsearch on logstash1002 |
[production] |
| 23:18 |
<bd808> |
scap by spage looks hung with 50 hosts not finished rsyncing |
[production] |
| 22:38 |
<bd808> |
Manually ran sync-common and scap-rebuild-cdbs on mw1163 |
[production] |
| 22:33 |
<mutante> |
depooling mw1163 |
[production] |
| 22:28 |
<ori> |
synchronized wmf-config/mc.php 'Set serializer to php for production memcache' |
[production] |
| 22:12 |
<spage> |
Started scap: Add Popups extension (Hovercards Beta Feature) to wmf17 and 18, not enabled yet |
[production] |
| 22:11 |
<spage> |
updated /a/common to {{Gerrit|I3d59b8246}}: Add new Popups extension to list |
[production] |
| 20:45 |
<subbu> |
deployed Parsoid ff8c49e9 with deploy 20923176 (a better log message then the previous one logged at 20:40) |
[production] |
| 20:40 |
<subbu> |
Deployed parsoid/deploy 20923176 |
[production] |
| 20:39 |
<robh> |
parsoid service successfully restarted across hosts via dsh |
[production] |
| 20:06 |
<yurik> |
synchronized docroot/bits/WikipediaMobileFirefoxOS/ |
[production] |
| 20:03 |
<mutante> |
shut down sq67-70 |
[production] |
| 20:00 |
<akosiaris> |
disabled puppet on labsdb1004 until osm2pgsql completes |
[production] |
| 19:21 |
<mutante> |
sq67-70: disable puppet,revoke puppet certs,delete salt keys and stored configs (delete from icinga) |
[production] |
| 19:15 |
<mutante> |
shutting down ssl1-4 |
[production] |
| 18:59 |
<bblack> |
all varnishes upgraded to 3.0.5plus-wmftest-wm5 (without restart, just adds header vmod) |
[production] |
| 18:46 |
<mutante> |
ssl1-4: puppet agent --disable, puppetstoredconfigclean, revoking puppet certs |
[production] |
| 18:24 |
<cmjohnson1> |
powering down mw1183 to reseat dimm |
[production] |
| 17:56 |
<cmjohnson1> |
mw1177 powering down to reseat DIMM |
[production] |