|
2014-03-20
§
|
| 19:23 |
<reedy> |
Finished scap: Rebuild 1.23wmf19 l10n cache for wikibase (duration: 12m 01s) |
[production] |
| 19:10 |
<reedy> |
Started scap: Rebuild 1.23wmf19 l10n cache for wikibase |
[production] |
| 19:08 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.23wmf19 |
[production] |
| 19:02 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.23wmf18 |
[production] |
| 19:00 |
<mutante> |
disk full on zirconium - gzipping some etherpadlite.sql dump i found |
[production] |
| 17:41 |
<reedy> |
synchronized php-1.23wmf19 'Update Wikidata and WikimediaMessages' |
[production] |
| 17:06 |
<Krinkle> |
Reloading Zuul to deploy Ie800ed90b51c47d5a1 |
[production] |
| 16:58 |
<mutante> |
repooling mw1163 (it's back in dsh as well) |
[production] |
| 16:38 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.23wmf18 till window |
[production] |
| 16:05 |
<bd808> |
Ran /usr/local/bin/sync-common && /usr/local/bin/scap-rebuild-cdbs on mw1163. Should not repool until it's back in the dsh group. Should me manually synced just before repooling. |
[production] |
| 15:47 |
<hashar> |
fixed salt-minion service on deployment-cache-upload01 and deployment-cache-mobile03 by deleting /etc/salt/pki/minion/minion_master.pub |
[releng] |
| 15:36 |
<reedy> |
Finished scap: testwiki to 1.23wmf19 and build l10n cache (duration: 15m 32s) |
[production] |
| 15:30 |
<hashar> |
migrated deployment-cache-upload01.eqiad.wmflabs and deployment-cache-mobile03.eqiad.wmflabs to use the salt/puppetmaster deployment-salt.eqiad.wmflabs. |
[releng] |
| 15:30 |
<hashar> |
deployment-cache-upload01.eqiad.wmflabs and deployment-cache-mobile03.eqiad.wmflabs recovered!! /dev/vdb does not exist on eqiad which caused the instance to be stalled. |
[releng] |
| 15:21 |
<reedy> |
Started scap: testwiki to 1.23wmf19 and build l10n cache |
[production] |
| 15:03 |
<maxsem> |
synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/118328' |
[production] |
| 15:00 |
<bd808> |
logstash stopped ingesting logs at 2014-03-19T22:37:54.000Z. |
[production] |
| 14:57 |
<bd808> |
restarted logstash on logstash1001.eqiad.wmnet |
[production] |
| 10:48 |
<hashar> |
Stopped the simplewiki script. Would need to recreate the db from scratch instead |
[releng] |
| 10:37 |
<hashar> |
Cleaning up simplewiki by deleting most pages in the main namespace. Would free up some disk space. deleteBatch.php is running in a screen on deployment-bastion.pmtpa.wmflabs |
[releng] |
| 10:08 |
<hashar> |
applying role::labs::lvm::mnt on deployment-db1 to provide additional disk space on /mnt |
[releng] |
| 09:39 |
<hashar> |
convert all remaining hosts but db1 to use the local puppet and salt masters |
[releng] |
| 03:23 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Mar 20 03:23:37 UTC 2014 (duration 23m 35s) |
[production] |
| 02:32 |
<LocalisationUpdate> |
completed (1.23wmf18) at 2014-03-20 02:32:10+00:00 |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.23wmf17) at 2014-03-20 02:17:34+00:00 |
[production] |
| 01:26 |
<Tim> |
gave Sam Reed access to racktables |
[production] |
| 00:26 |
<bd808> |
scap-slave rsync servers have "hosts allow = 10.0.0.0/16 10.64.0.0/22 10.64.16.0/22 10.64.32.0/22 208.80.152.0/22"; missing new 10.64.48.0/22 subnet |
[production] |
| 00:21 |
<bd808> |
Subnet for row D still not in rsync server config? |
[production] |
| 00:21 |
<bd808> |
scaps failed because "@ERROR: access denied to common from mw1202.eqiad.wmnet (10.64.48.34)" |
[production] |
| 00:16 |
<bd808> |
finished dsh -c -M -m mw1201,mw1202,mw1203,mw1208,mw1209,mw1210 -- '/usr/local/bin/sync-common; /usr/local/bin/scap-rebuild-cdbs' |
[production] |
| 00:13 |
<bd808> |
dsh -c -M -m mw1201,mw1202,mw1203,mw1208,mw1209,mw1210 -- '/usr/local/bin/sync-common; /usr/local/bin/scap-rebuild-cdbs' |
[production] |
|
2014-03-19
§
|
| 23:57 |
<spage> |
synchronized wmf-config 'Config change to enable extension Popups (Hovercards) on mediawikiwiki' |
[production] |
| 23:30 |
<spage> |
Finished scap: Retry Add Popups extension (Hovercards Beta Feature) to wmf17 and 18, not enabled yet (duration: 07m 15s) |
[production] |
| 23:22 |
<spage> |
Started scap: Retry Add Popups extension (Hovercards Beta Feature) to wmf17 and 18, not enabled yet |
[production] |
| 23:22 |
<spage> |
scap aborted: Add Popups extension (Hovercards Beta Feature) to wmf17 and 18, not enabled yet (duration: 69m 26s) |
[production] |
| 23:20 |
<bd808> |
restarted elasticsearch on logstash1002 |
[production] |
| 23:18 |
<bd808> |
scap by spage looks hung with 50 hosts not finished rsyncing |
[production] |
| 22:38 |
<bd808> |
Manually ran sync-common and scap-rebuild-cdbs on mw1163 |
[production] |
| 22:33 |
<mutante> |
depooling mw1163 |
[production] |
| 22:28 |
<ori> |
synchronized wmf-config/mc.php 'Set serializer to php for production memcache' |
[production] |
| 22:12 |
<spage> |
Started scap: Add Popups extension (Hovercards Beta Feature) to wmf17 and 18, not enabled yet |
[production] |
| 22:11 |
<spage> |
updated /a/common to {{Gerrit|I3d59b8246}}: Add new Popups extension to list |
[production] |
| 21:23 |
<bd808> |
Converted deployment-cache-text02 to use local puppet & salt masters |
[releng] |
| 20:45 |
<subbu> |
deployed Parsoid ff8c49e9 with deploy 20923176 (a better log message then the previous one logged at 20:40) |
[production] |
| 20:40 |
<subbu> |
Deployed parsoid/deploy 20923176 |
[production] |
| 20:39 |
<robh> |
parsoid service successfully restarted across hosts via dsh |
[production] |
| 20:21 |
<hashar> |
migrating eqiad varnish caches to use xfs |
[releng] |
| 20:06 |
<yurik> |
synchronized docroot/bits/WikipediaMobileFirefoxOS/ |
[production] |
| 20:03 |
<mutante> |
shut down sq67-70 |
[production] |
| 20:00 |
<akosiaris> |
disabled puppet on labsdb1004 until osm2pgsql completes |
[production] |