|
2016-12-29
§
|
| 23:19 |
<akosiaris> |
schedule downtime for ferm checks on kubernetes nodes. Some race between kubernetes + ferm, investigating |
[production] |
| 21:19 |
<otto@tin> |
Finished deploy [eventstreams/deploy@4098bb4]: (no message) (duration: 01m 59s) |
[production] |
| 21:17 |
<otto@tin> |
Starting deploy [eventstreams/deploy@4098bb4]: (no message) |
[production] |
| 16:05 |
<_joe_> |
restarted HHVM on mw1279, stuck in HPHP::Treadmill::getAgeOldestRequest |
[production] |
| 15:56 |
<akosiaris> |
merging https://gerrit.wikimedia.org/r/329597 for T154278 (IP throttle raise) |
[production] |
| 15:55 |
<akosiaris@tin> |
Synchronized wmf-config/throttle.php: (no message) (duration: 00m 42s) |
[production] |
| 12:20 |
<akosiaris> |
running sudo apt-get autoremove on labtestnet2001. Removing various older kernels |
[production] |
| 12:19 |
<akosiaris> |
running sudo apt-get autoremove on labtestnet2001 |
[production] |
| 05:22 |
<cwd> |
updated civicrm from 038e166269667e7b1c0e9ef54b06ae79b546c76e to f78c894ba6686f0b512e8120eb75e928ef8c92fe |
[production] |
| 01:10 |
<cwd> |
updated payments-wiki from 5b4ced90352ec2c51186f93946840af24eaef290 to 21ea9bc4a46c96f84685b33ef06fd1b4c40e797e |
[production] |
| 00:18 |
<cwd> |
updated smash-pig from c3eaadb737a02db27f3e97a8e60981c7182e877c to 3da597fdfc9718d10fae9c8e94825729a561d890 |
[production] |
|
2016-12-28
§
|
| 23:13 |
<cwd> |
rolled back smashpig |
[production] |
| 22:57 |
<cwd> |
updated smashpig from c3eaadb737a02db27f3e97a8e60981c7182e877c to 8597bd45fae3710bcb7c9bd0c592641723e52511 |
[production] |
| 14:12 |
<mobrovac@tin> |
Finished deploy [trending-edits/deploy@c5d239b]: Add depooling and repooling to Trending Edits T144602 (duration: 01m 21s) |
[production] |
| 14:11 |
<mobrovac@tin> |
Starting deploy [trending-edits/deploy@c5d239b]: Add depooling and repooling to Trending Edits T144602 |
[production] |
| 14:07 |
<mobrovac@tin> |
Finished deploy [mobileapps/deploy@ae22656]: Add depooling and repooling to MCS T144602 (duration: 02m 34s) |
[production] |
| 14:05 |
<mobrovac@tin> |
Starting deploy [mobileapps/deploy@ae22656]: Add depooling and repooling to MCS T144602 |
[production] |
| 14:00 |
<mobrovac@tin> |
Finished deploy [electron-render/deploy@b2a820e]: Add depooling and repooling to PDFRender T144602 (duration: 01m 26s) |
[production] |
| 13:59 |
<mobrovac@tin> |
Starting deploy [electron-render/deploy@b2a820e]: Add depooling and repooling to PDFRender T144602 |
[production] |
| 13:51 |
<mobrovac@tin> |
Finished deploy [graphoid/deploy@151f26c]: Add depooling and repooling to Graphoid T144602 (duration: 01m 37s) |
[production] |
| 13:49 |
<mobrovac@tin> |
Starting deploy [graphoid/deploy@151f26c]: Add depooling and repooling to Graphoid T144602 |
[production] |
| 13:44 |
<mobrovac@tin> |
Finished deploy [cxserver/deploy@0279029]: Add depooling and repooling to CXServer T144602 (duration: 03m 22s) |
[production] |
| 13:41 |
<mobrovac@tin> |
Starting deploy [cxserver/deploy@0279029]: Add depooling and repooling to CXServer T144602 |
[production] |
| 13:41 |
<mobrovac@tin> |
Finished deploy [cxserver/deploy@0279029]: (no message) (duration: 00m 25s) |
[production] |
| 13:40 |
<mobrovac@tin> |
Starting deploy [cxserver/deploy@0279029]: (no message) |
[production] |
| 13:33 |
<mobrovac@tin> |
Finished deploy [citoid/deploy@da96f4b]: Add depooling and repooling to Citoid T144602 (duration: 01m 59s) |
[production] |
| 13:31 |
<mobrovac@tin> |
Starting deploy [citoid/deploy@da96f4b]: Add depooling and repooling to Citoid T144602 |
[production] |
| 13:21 |
<mobrovac@tin> |
Finished deploy [mathoid/deploy@79fdd56]: Add depooling and repooling to Mathoid T144602 (duration: 02m 31s) |
[production] |
| 13:19 |
<mobrovac@tin> |
Starting deploy [mathoid/deploy@79fdd56]: Add depooling and repooling to Mathoid T144602 |
[production] |
| 07:36 |
<legoktm> |
added reedy to "integration" gerrit group per T154207 |
[production] |
| 04:15 |
<awight> |
disable Fundraising listeners via extraordinary workaround: from f14337880e389f95f791c11ab17dfabf36f5317f to 6b864916a2b09ca30dfa17476dbc0a74b0aa7828 |
[production] |
| 03:52 |
<awight> |
disabled Fundraising audit parsers |
[production] |
| 03:43 |
<awight> |
put payments frontends into maintenance mode |
[production] |
| 02:47 |
<awight> |
reenabling Fundraisin thank-you job |
[production] |
| 02:46 |
<awight> |
update Fundraising CiviCRM from 454679d201ed59c76a3905cf3ad5ee2d14fef93f to 038e166269667e7b1c0e9ef54b06ae79b546c76e |
[production] |
| 01:34 |
<awight> |
disabled thank-you mailer per T154209 |
[production] |
|
2016-12-27
§
|
| 21:32 |
<awight> |
reenabled WMF Fundraising low-level campaigns |
[production] |
| 21:16 |
<awight> |
reenabled all Fundraising jobs except for dedupe |
[production] |
| 16:23 |
<_joe_> |
power down prometheus2003 |
[production] |
| 05:44 |
<Amir1> |
ladsgroup@terbium:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=wikidatawiki (T154168) |
[production] |
| 05:25 |
<Amir1> |
finished deploy of ores:228b9b4 in all nodes (T154168) |
[production] |
| 05:25 |
<ladsgroup@tin> |
Finished deploy [ores/deploy@228b9b4]: (no message) (duration: 18m 46s) |
[production] |
| 05:14 |
<Amir1> |
starting deploy of ores:228b9b4 in all nodes (T154168) |
[production] |
| 05:06 |
<ladsgroup@tin> |
Starting deploy [ores/deploy@228b9b4]: (no message) |
[production] |
| 05:06 |
<Amir1> |
starting deploy of ores:228b9b4 in canary nodes (T154168) |
[production] |