|
2015-02-04
§
|
| 14:10 |
<godog> |
bounce navtiming on hafnium to pick up dns changes |
[production] |
| 12:42 |
<godog> |
stop bacula-fd on tungsten, backups running during migration |
[production] |
| 12:41 |
<_joe_> |
installing the new HHVM package on jobrunners |
[production] |
| 12:28 |
<godog> |
bounce txstatsd on ms-fe* |
[production] |
| 12:28 |
<godog> |
bounce txstatsd on ms-be* |
[production] |
| 12:00 |
<godog> |
bounce diamond in batches in ulsfo |
[production] |
| 11:57 |
<godog> |
bounce diamond in batches in esams |
[production] |
| 11:51 |
<godog> |
bounce mwprof on tungsten to force picking up dns changes |
[production] |
| 11:35 |
<_joe_> |
installing the new hhvm package on api, one at a time |
[production] |
| 11:23 |
<godog> |
start migrating graphite from tungsten to graphite1001 https://gerrit.wikimedia.org/r/#/c/188036/1 https://gerrit.wikimedia.org/r/#/c/188035/1 https://phabricator.wikimedia.org/T85909 |
[production] |
| 10:14 |
<ori> |
Finished scap: I78446aacb: [Regression] Revert "Non wikipedias to 1.25wmf15" (duration: 31m 34s) |
[production] |
| 10:06 |
<godog> |
start migrating graphite from tungsten to graphite1001 https://gerrit.wikimedia.org/r/#/c/188036/1 https://gerrit.wikimedia.org/r/#/c/188035/1 https://phabricator.wikimedia.org/T85909 |
[production] |
| 10:06 |
<ori> |
restarted hung HHVM on mw1039 |
[production] |
| 09:42 |
<ori> |
Started scap: I78446aacb: [Regression] Revert "Non wikipedias to 1.25wmf15" |
[production] |
| 09:42 |
<ori> |
scap aborted: I78446aacb: [Regression] Revert "Non wikipedias to 1.25wmf15" (duration: 00m 02s) |
[production] |
| 09:42 |
<ori> |
Started scap: I78446aacb: [Regression] Revert "Non wikipedias to 1.25wmf15" |
[production] |
| 08:57 |
<_joe_> |
installing the new hhvm package on all appservers, one at a time |
[production] |
| 07:49 |
<qchris> |
Manual failover of Hadoop namenode from analytics1002 to analytics1001, as analytics1002 had Heap space errors |
[production] |
| 05:16 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1057 (duration: 00m 06s) |
[production] |
| 04:49 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Feb 4 04:48:42 UTC 2015 (duration 48m 41s) |
[production] |
| 02:48 |
<tstarling> |
Synchronized php-1.25wmf15/includes/specials/SpecialUserrights.php: Unbreak interwiki user rights granting (duration: 00m 05s) |
[production] |
| 02:30 |
<LocalisationUpdate> |
completed (1.25wmf15) at 2015-02-04 02:29:17+00:00 |
[production] |
| 02:29 |
<l10nupdate> |
Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s) |
[production] |
| 02:16 |
<LocalisationUpdate> |
completed (1.25wmf14) at 2015-02-04 02:15:35+00:00 |
[production] |
| 02:15 |
<l10nupdate> |
Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 03s) |
[production] |
| 01:47 |
<bd808> |
restarted elasticsearch on logstash1001; rolling restart part 3 of 3 |
[production] |
| 01:39 |
<bd808> |
restarted elasticsearch on logstash1002; rolling restart of cluster part 2 of 3 |
[production] |
| 00:38 |
<robh> |
replacing dumps.w.o sha1 cert with sha256 |
[production] |
|
2015-02-03
§
|
| 22:54 |
<bd808> |
restarted elasticsearch on logstash1003 |
[production] |
| 22:53 |
<bd808> |
starting rolling restart of logstash elasticsearch cluster to pick up index.merge.scheduler.max_thread_count puppet change |
[production] |
| 22:52 |
<robh> |
magnesium apache reload for rt cert replacement |
[production] |
| 22:43 |
<robh> |
replacing etherpad sha1 with sha256 cert |
[production] |
| 22:13 |
<reedy> |
Synchronized php-1.25wmf15/extensions/WikimediaMaintenance: tmp script (duration: 00m 07s) |
[production] |
| 21:38 |
<maxsem> |
Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/188300/ (duration: 00m 07s) |
[production] |
| 21:32 |
<bblack> |
repooled cp10[67]0 |
[production] |
| 21:29 |
<mutante> |
restarted gitblit |
[production] |
| 21:17 |
<andrewbogott> |
increased opendj lookthrough-limit to 12000 on both ldap hosts. We just hit lucky 5000 users and some queries stopped working. |
[production] |
| 20:59 |
<bblack> |
depooling cp1060, cp1070 (1 each bits + mobile) for reinstall |
[production] |
| 20:59 |
<robh> |
updating gerrit.wikimedia.org cert |
[production] |
| 19:30 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.25wmf15 |
[production] |
| 18:17 |
<YuviPanda> |
set up new images for ubuntu trusty / precise on labs, for https://phabricator.wikimedia.org/T87003 |
[production] |
| 17:23 |
<bblack> |
repooled cp1064 |
[production] |
| 17:07 |
<bd808> |
restarted elasticsearch on logstash1002; OOM |
[production] |
| 16:55 |
<bblack> |
cp1064 (eqiad upload cache) depooled in pybal |
[production] |
| 15:39 |
<_joe_> |
installing a new package to canary servers |
[production] |
| 14:55 |
<_joe_> |
uploaded a new hhvm package version, deploying to testwiki and beta |
[production] |
| 08:50 |
<reedy> |
Synchronized wmf-config/: Bye bye Solarium (duration: 00m 06s) |
[production] |
| 08:20 |
<ori> |
Synchronized wmf-config/InitialiseSettings.php: Ie7b32e3d8: Add log group for T87645 (duration: 00m 05s) |
[production] |
| 08:19 |
<ori> |
Synchronized php-1.25wmf14/includes/EditPage.php: Id376f9e75: Hack for T87645, since maybe it is still happening (duration: 00m 07s) |
[production] |
| 08:17 |
<ori> |
Synchronized php-1.25wmf15/includes/EditPage.php: Id376f9e75: Hack for T87645, since maybe it is still happening (duration: 00m 05s) |
[production] |