|
2015-01-17
§
|
| 06:36 |
<YuviPanda> |
restarted dnsmasq on labnet1001 (see https://wikitech.wikimedia.org/wiki/Labs_DNS#DHCP_and_internal_DNS for how to) |
[production] |
| 04:47 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sat Jan 17 04:47:42 UTC 2015 (duration 47m 41s) |
[production] |
| 02:31 |
<LocalisationUpdate> |
completed (1.25wmf15) at 2015-01-17 02:30:57+00:00 |
[production] |
| 02:30 |
<l10nupdate> |
Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s) |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.25wmf14) at 2015-01-17 02:18:24+00:00 |
[production] |
| 02:18 |
<l10nupdate> |
Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s) |
[production] |
| 02:17 |
<ori> |
mw1148: threads in stuck in __lll_lock_wait (); restarted HHVM. |
[production] |
| 02:16 |
<krinkle> |
Synchronized php-1.25wmf14/includes/content/JsonContent.php: Ic1d10393912fcefa22d (duration: 00m 06s) |
[production] |
| 02:16 |
<krinkle> |
Synchronized php-1.25wmf14/resources/src/mediawiki/mediawiki.content.json.css: Ic1d10393912fcefa22d (duration: 00m 06s) |
[production] |
| 02:14 |
<krinkle> |
Synchronized php-1.25wmf15/includes/content/JsonContent.php: Ic1d10393912fcefa22d (duration: 00m 05s) |
[production] |
| 02:14 |
<krinkle> |
Synchronized php-1.25wmf15/resources/src/mediawiki/mediawiki.content.json.css: Ic1d10393912fcefa22d (duration: 00m 06s) |
[production] |
| 02:10 |
<ori> |
Synchronized wmf-config/StartProfiler.php: I4e3871d3d: xenon: Annotate file scope and closure scope with filename (duration: 00m 05s) |
[production] |
| 00:13 |
<bd808> |
restarted elasticserch on logstash1001 & logstash1003; OOM |
[production] |
|
2015-01-16
§
|
| 23:33 |
<bd808> |
ran `LTRIM logstash -50000 9999999` on redis queues to drop ~4M events in backlog |
[production] |
| 22:14 |
<bd808> |
restarted elasticsearch on logstash1001; OOM errors |
[production] |
| 21:21 |
<bd808> |
restarted elasticsearch on logstash1001 |
[production] |
| 21:18 |
<marktraceur> |
Finished scap: Fix UploadWizard regression and EventLogging errors (duration: 31m 06s) |
[production] |
| 21:17 |
<bd808> |
OOM for elasticsearch on logstash1001 caused a dropped shard and icinga alerts |
[production] |
| 20:47 |
<marktraceur> |
Started scap: Fix UploadWizard regression and EventLogging errors |
[production] |
| 20:17 |
<bd808> |
Synchronized wmf-config/InitialiseSettings.php: Allow wgDebugLogGroups to exclude logstash append (e808e690) (duration: 00m 05s) |
[production] |
| 20:17 |
<bd808> |
Synchronized wmf-config/logging.php: Allow wgDebugLogGroups to exclude logstash append (e808e690) (duration: 00m 07s) |
[production] |
| 18:13 |
<bd808> |
document count not changing for logstash-2015.01.16 index |
[production] |
| 17:59 |
<bd808> |
Synchronized wmf-config/logging-labs.php: beta: Allow wgDebugLogGroups to exclude logstash append (03c3ab27) (duration: 00m 06s) |
[production] |
| 17:50 |
<bblack> |
depooled amssq42 text cache in esams |
[production] |
| 17:44 |
<ejegg> |
updated tools from 88b57fea517d2232e8ae906df550f426b6574f24 to 84442d51a841af4265ff103827cda83d5dd9dc54 |
[production] |
| 17:24 |
<demon> |
Synchronized wmf-config/: (no message) (duration: 00m 05s) |
[production] |
| 17:21 |
<ejegg> |
updated civicrm from d648ededf5c9fc2b0ebf989300ca2037956418e3 to 4fa10ec9e3afbf65e6cbd523138cdc4b4485c482 |
[production] |
| 17:17 |
<demon> |
Synchronized wmf-config/: (no message) (duration: 00m 06s) |
[production] |
| 16:48 |
<ottomata> |
finished hadoop namenode migration. Hadoop cluster is back online |
[production] |
| 16:48 |
<bd808> |
Upgraded elasticsearch and restarted on all logstash nodes |
[production] |
| 16:43 |
<bd808> |
shutdown whole elasticsearch cluster for logstash |
[production] |
| 16:39 |
<bd808> |
restarted elasticsearch on logstash1001 |
[production] |
| 16:07 |
<ottomata> |
stopping hadoop cluster |
[production] |
| 08:01 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1051 db1056, warm up (duration: 00m 10s) |
[production] |
| 05:54 |
<ori> |
<jgage> mtr shows me packet loss between cr2-eqiad.wikimedia.org and 206.126.236.21 aka eqixva-google-gige.google.com |
[production] |
| 04:40 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Jan 16 04:40:10 UTC 2015 (duration 40m 9s) |
[production] |
| 04:22 |
<Tim> |
on mw1228 doing some tests to figure out why incorrect Expires header is being sent on requests for /images/* |
[production] |
| 03:09 |
<ori> |
Synchronized php-1.25wmf14/includes/content/JsonContent.php: I2f4f9cb343: Let subclasses specify content model in JsonContent (duration: 00m 06s) |
[production] |
| 03:01 |
<springle> |
xtrabackup clone db1020 to db1046 |
[production] |
| 02:31 |
<LocalisationUpdate> |
completed (1.25wmf15) at 2015-01-16 02:31:37+00:00 |
[production] |
| 02:31 |
<l10nupdate> |
Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s) |
[production] |
| 02:19 |
<LocalisationUpdate> |
completed (1.25wmf14) at 2015-01-16 02:19:04+00:00 |
[production] |
| 02:19 |
<l10nupdate> |
Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s) |
[production] |
| 02:06 |
<ori> |
EventLogging syncs were of I335ad42bb: JsonSchemaContent: Fix html rendering of objects and arrays |
[production] |
| 02:03 |
<ori> |
Synchronized php-1.25wmf14/extensions/EventLogging: (no message) (duration: 00m 05s) |
[production] |
| 02:03 |
<ori> |
Synchronized php-1.25wmf15/extensions/EventLogging: (no message) (duration: 00m 06s) |
[production] |
| 00:47 |
<mutante> |
on both puppetmasters: chown gitpuppet /var/lib/git/operations/puppet/.git/logs/refs/heads/production & .git/logs/HEAD & .git/logs/refs/remotes/origin to fix puppet-merge. git pulled on strontium |
[production] |
| 00:46 |
<mutante> |
restarted morebots |
[production] |