|
2017-06-06
ยง
|
| 20:21 |
<RainbowSprinkles> |
gerrit: Down for just a moment, finally doing point release on cobalt |
[production] |
| 19:57 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.4 |
[production] |
| 19:45 |
<demon@tin> |
Finished scap: testwiki to wmf.4 + prepping l10n. again (x2) (duration: 20m 25s) |
[production] |
| 19:36 |
<mutante> |
cobalt - removed systemd unit file (that has issues with ulimit and isn't used yet) - ran "systemctl reset-failed" which cleared the "systemctl status" which made the Icinga check recover |
[production] |
| 19:24 |
<demon@tin> |
Started scap: testwiki to wmf.4 + prepping l10n. again (x2) |
[production] |
| 19:23 |
<demon@tin> |
scap failed: RuntimeError scap failed: average error rate on 1/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/3888cca979647b9381a7739b0bdbc88e for details) (duration: 13m 32s) |
[production] |
| 19:23 |
<demon@tin> |
scap failed: average error rate on 1/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/3888cca979647b9381a7739b0bdbc88e for details) |
[production] |
| 19:10 |
<demon@tin> |
Started scap: testwiki to wmf.4 + prepping l10n. again |
[production] |
| 19:08 |
<demon@tin> |
Synchronized README: No-op, just forcing co-master sync (duration: 01m 27s) |
[production] |
| 19:01 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: testwiki back to wmf.2 |
[production] |
| 18:55 |
<maxsem@tin> |
Finished scap: LoginNotify to testwiki - rebuild messages (duration: 38m 19s) |
[production] |
| 18:16 |
<maxsem@tin> |
Started scap: LoginNotify to testwiki - rebuild messages |
[production] |
| 18:15 |
<maxsem@tin> |
Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/357317/2 (duration: 00m 44s) |
[production] |
| 18:10 |
<maxsem@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/357317/2 (duration: 00m 44s) |
[production] |
| 18:03 |
<demon@tin> |
Finished scap: testwiki to wmf.3, prepping l10n cache (duration: 31m 58s) |
[production] |
| 17:31 |
<demon@tin> |
Started scap: testwiki to wmf.3, prepping l10n cache |
[production] |
| 16:53 |
<moritzm> |
installing wireshark security updates on trusty (jessie already fixed) |
[production] |
| 16:41 |
<bblack> |
rebooted lvs1007 (kernel update) |
[production] |
| 16:35 |
<bblack> |
rebooted lvs1007 (kernel update) |
[production] |
| 15:21 |
<otto@tin> |
Finished deploy [eventlogging/analytics@37233cd]: (no justification provided) (duration: 00m 04s) |
[production] |
| 15:21 |
<otto@tin> |
Started deploy [eventlogging/analytics@37233cd]: (no justification provided) |
[production] |
| 14:58 |
<moritzm> |
installing libsndfile security updates on trusty |
[production] |
| 14:01 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Restore db1094 original weight (duration: 00m 40s) |
[production] |
| 13:46 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1094 weight (duration: 00m 40s) |
[production] |
| 13:39 |
<elukey> |
shutdown analytics1033 and analytics1039 to replace their BBU - T166140 |
[production] |
| 13:33 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1094 with low weight (duration: 00m 40s) |
[production] |
| 12:58 |
<marostegui> |
Shutdown db1094 for maintenance - T166518 |
[production] |
| 12:58 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1094 for maintenance - T166518 (duration: 00m 39s) |
[production] |
| 12:51 |
<godog> |
upgrade scap to 3.5.8 - T127762 |
[production] |
| 12:41 |
<mobrovac@tin> |
Finished deploy [changeprop/deploy@e92dd66]: Bump src to bc8abf3 (duration: 01m 45s) |
[production] |
| 12:40 |
<mobrovac@tin> |
Started deploy [changeprop/deploy@e92dd66]: Bump src to bc8abf3 |
[production] |
| 12:16 |
<bblack> |
cp1049 - restaret varnish backend for mailbox lag |
[production] |
| 12:08 |
<gehel> |
kill stuck osm replication on maps1001 |
[production] |
| 11:28 |
<akosiaris@tin> |
Finished deploy [servermon/servermon@4a2288f]: (no justification provided) (duration: 00m 04s) |
[production] |
| 11:28 |
<akosiaris@tin> |
Started deploy [servermon/servermon@4a2288f]: (no justification provided) |
[production] |
| 11:16 |
<moritzm> |
uploaded ferm 2.3.2+wmf1 to apt.wikimedia.org/stretch-wikimedia (T166653) |
[production] |
| 11:02 |
<ladsgroup@tin> |
Synchronized wmf-config/Wikibase-production.php: Enabling writing in full entity id in testwikidatawiki (T165197) (duration: 00m 39s) |
[production] |
| 10:22 |
<moritzm> |
installing NSS security updates |
[production] |
| 09:43 |
<moritzm> |
installing perl security updates |
[production] |
| 09:41 |
<akosiaris> |
stop jobchron/jobrunner processes across jobrunner and videoscalers in codfw |
[production] |
| 09:35 |
<akosiaris> |
restart jobchron service across videoscalers T129148 |
[production] |
| 09:33 |
<akosiaris> |
restart jobchron service across jobrunners T129148 |
[production] |
| 09:32 |
<akosiaris@tin> |
Finished deploy [jobrunner/jobrunner@161c84c]: (no justification provided) (duration: 01m 17s) |
[production] |
| 09:31 |
<akosiaris@tin> |
Started deploy [jobrunner/jobrunner@161c84c]: (no justification provided) |
[production] |
| 09:29 |
<akosiaris> |
running puppet on jobrunners T129148 |
[production] |
| 09:25 |
<akosiaris> |
running puppet on videoscalers T129148 |
[production] |
| 09:25 |
<akosiaris> |
moving around jobrunner/jobrunner was probably not required T129148 |
[production] |
| 09:19 |
<akosiaris> |
running puppet again on tin, after moving /serv/deployment/jobrunner/jobrunner T129148 |
[production] |
| 09:12 |
<akosiaris> |
running puppet on mw1161 T129148 |
[production] |
| 09:11 |
<akosiaris> |
git pull and scap deploy --init for jobrunner T129148 |
[production] |