|
2015-10-26
§
|
| 20:21 |
<ebernhardson> |
started copy of eqiad elasticsearch indices to noeblium |
[production] |
| 20:19 |
<YuviPanda> |
stress test on nobelium complete, CPU temperature didn't go above 65C |
[production] |
| 20:13 |
<paravoid> |
deactivating ulsfo<->NTT BGP peering due to upcoming network migration |
[production] |
| 20:05 |
<YuviPanda> |
running stress on nobelium |
[production] |
| 19:36 |
<cmjohnson1> |
swapped bad disk on db1030 |
[production] |
| 15:44 |
<ebernhardson@tin> |
Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEvents.php: Re-deploy WME changes after deploying necessary CirrusSearch change first (duration: 00m 18s) |
[production] |
| 15:43 |
<ebernhardson@tin> |
Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/modules: Re-deploy WME changes after deploying necessary CirrusSearch change first (duration: 00m 17s) |
[production] |
| 15:30 |
<ebernhardson@tin> |
Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Undeploy eventlogging search schema from CirrusSearch (duration: 00m 18s) |
[production] |
| 15:21 |
<ebernhardson@tin> |
Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents: rollback (duration: 00m 18s) |
[production] |
| 15:13 |
<ebernhardson@tin> |
Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEvents.php: Move search schema from cirrussearch -> wikimediavents (duration: 00m 19s) |
[production] |
| 15:13 |
<ebernhardson@tin> |
Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/modules/: Move search schema from cirrussearch -> wikimediavents (duration: 00m 17s) |
[production] |
| 15:06 |
<ebernhardson@tin> |
Synchronized php-1.27.0-wmf.3/extensions/WikimediaEvents/WikimediaEvents.php: Update satisfaction schema id due to bad varnish caching of old id (duration: 00m 17s) |
[production] |
| 15:04 |
<ebernhardson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Remove Predundant Page and Index namespaces from $wgContentNamespaces (duration: 00m 17s) |
[production] |
| 14:58 |
<bblack> |
repooling cp1059 varnish mobile frontend (wiped) |
[production] |
| 14:18 |
<aude@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable GeoData on Wikidata (duration: 00m 17s) |
[production] |
| 14:09 |
<aude@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable geosearch on test.wikidata (duration: 00m 17s) |
[production] |
| 13:32 |
<aude@tin> |
Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Add justMapping option to updateOneSearchIndexConfig script (updated submodule) (duration: 00m 18s) |
[production] |
| 13:26 |
<aude@tin> |
Synchronized php-1.27.0-wmf.3/extensions/CirrusSearch: Add justMapping option to updateOneSearchIndexConfig script (duration: 00m 18s) |
[production] |
| 09:42 |
<dcausse> |
deleting unused elasticsearch indices in eqiad (T112863) |
[production] |
| 09:20 |
<_joe_> |
restarting etcd on conf1001 |
[production] |
| 09:16 |
<jynus> |
rebooting and installing jessie on db2060-db2070 |
[production] |
| 05:48 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Mon Oct 26 05:48:19 UTC 2015 (duration 48m 18s) |
[production] |
| 02:31 |
<l10nupdate@tin> |
LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-26 02:31:46+00:00 |
[production] |
| 02:27 |
<l10nupdate@tin> |
Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 14s) |
[production] |
|
2015-10-24
§
|
| 19:34 |
<twentyafterfour> |
deployed https://gerrit.wikimedia.org/r/#/c/248638/ and restarted apache on iridium |
[production] |
| 06:00 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sat Oct 24 06:00:53 UTC 2015 (duration 0m 52s) |
[production] |
| 05:44 |
<ori@tin> |
Synchronized php-1.27.0-wmf.3/extensions/Flow/includes/Data/Index/FeatureIndex.php: (no message) (duration: 00m 17s) |
[production] |
| 05:11 |
<hoo> |
Set an email address for user "Ymnes", after request. Confirmed by several, including. |
[production] |
| 02:37 |
<l10nupdate@tin> |
LocalisationUpdate completed (1.27.0-wmf.3) at 2015-10-24 02:37:34+00:00 |
[production] |
| 02:33 |
<l10nupdate@tin> |
Synchronized php-1.27.0-wmf.3/cache/l10n: l10nupdate for 1.27.0-wmf.3 (duration: 08m 23s) |
[production] |
| 00:21 |
<ebernhardson> |
started second machine (nobelium) performing copy of elasticsearch indices to codfw with 40 threads |
[production] |
|
2015-10-23
§
|
| 23:31 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki hewikisource |
[production] |
| 23:31 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki ukwiki |
[production] |
| 23:09 |
<hoo> |
Started a local rename for Stefan4 to Stefan2 on commons, per request (and attributed to) DerHexer |
[production] |
| 22:23 |
<robh> |
powercycled nobelium |
[production] |
| 21:24 |
<mutante> |
nobelium - powercycled, no console output |
[production] |
| 20:04 |
<jynus> |
rename user job had created lag on almost all enwiki dbs, things should be better now |
[production] |
| 19:08 |
<urandom> |
starting nodetool cleanup on restbase-test2001-b |
[production] |
| 19:07 |
<urandom> |
starting nodetool cleanup on restbase-test2001-a |
[production] |
| 18:16 |
<godog> |
bounce grafana-server on krypton |
[production] |
| 17:58 |
<YuviPanda> |
moved mounts around on nobelium, mounted bigger disk on /var/lib/elasticsearch |
[production] |
| 17:57 |
<godog> |
delete wikimedia-grid grafana dashboard (saved a copy first) heavy graphite queries |
[production] |
| 17:53 |
<YuviPanda> |
stopping elasticsearch on nobelium for https://phabricator.wikimedia.org/T114856 |
[production] |
| 17:13 |
<akosiaris> |
enable_notifications=0 in neon's icinga for a few mins while the storm dies down |
[production] |
| 16:47 |
<godog> |
stop puppet on graphite1001 and graphite-index cron, suspected root cause |
[production] |
| 16:26 |
<godog> |
uwsgi timing out while serving requests, bounce also carbon daemons |
[production] |
| 16:20 |
<godog> |
bounce uwsgi for graphite-web on graphite1001 |
[production] |