|
2017-01-06
§
|
| 18:08 |
<godog> |
force puppet run on eqiad cache_upload to switch thumbs to codfw |
[production] |
| 16:17 |
<godog> |
bounce swift-proxy on ms-fe100[123] leave ms-fe1004 for investigation |
[production] |
| 16:14 |
<hashar> |
Restarting Nodepool |
[production] |
| 16:05 |
<ema> |
wiping codfw caches T154758 |
[production] |
| 15:29 |
<cmjohnson1> |
powering off mw1239 to reseat DIMM |
[production] |
| 15:22 |
<papaul> |
elastic2025-elastic2036 - signing puppet certs, salt-key, initial run |
[production] |
| 14:53 |
<mark> |
papaul powercycled asw-a7-codfw 14:50 |
[production] |
| 14:16 |
<reedy@tin> |
Finished scap: Rebuild message cache for Echo api messages being missing T154110 (duration: 25m 00s) |
[production] |
| 13:51 |
<reedy@tin> |
Started scap: Rebuild message cache for Echo api messages being missing T154110 |
[production] |
| 10:00 |
<ariel@tin> |
Synchronized wmf-config/throttle.php: test, noop (duration: 02m 45s) |
[production] |
| 09:24 |
<paravoid> |
asw-a7-codfw is down, serial console unresponsive |
[production] |
| 09:12 |
<ariel@tin> |
Synchronized wmf-config/throttle.php: Adjust throttle rule for Maharashtra 'Edit Wikipedia' workshop (VNGIASS) (duration: 02m 46s) |
[production] |
| 07:08 |
<moritzm> |
installing crypto++ security updates on trusty hosts |
[production] |
| 04:37 |
<matt_flaschen> |
Finished FlowFixInconsistentBoards.php (production mode) on all wikis |
[production] |
| 04:27 |
<matt_flaschen> |
Started FlowFixInconsistentBoards.php (production mode) on all wikis |
[production] |
| 04:10 |
<mutante> |
Icinga now using Letsencrypt cert and all good |
[production] |
| 03:42 |
<mutante> |
icinga - debugging issue with cert change |
[production] |
| 03:05 |
<papaul> |
OS instalaltion on elastic2025-elastic2036 |
[production] |
| 02:34 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.7) (duration: 13m 23s) |
[production] |
| 01:27 |
<bd808> |
Restarted logstash on logstash1002 (T154732) |
[production] |
| 01:06 |
<Krinkle> |
stream.wikimedia.org problems - nginx responds with HTTP 502 Bad Gateway to most requests |
[production] |
| 01:01 |
<ostriches> |
gerrit: back up from upgrade |
[production] |
| 01:00 |
<ostriches> |
gerrit: down for upgrade |
[production] |
| 00:13 |
<mutante> |
analytics1036, ms-fe1003 - ran puppet to fix Icinga |
[production] |
| 00:11 |
<mutante> |
carbon - stopping ganglia-monitor-aggregator for good |
[production] |
|
2017-01-05
§
|
| 23:49 |
<mutante> |
rolling out exim4 upgrades (DSA 3747-1) on all remaining eqiad (all-eqiad) |
[production] |
| 23:46 |
<godog> |
fix root-owned files on puppetmaster1001:/var/lib/git/operations/private/ causing /srv/private post-commit hook to fail |
[production] |
| 23:18 |
<mutante> |
switching eqiad ganglia aggregator - running puppet on install1001 - disabling on carbon, re-enabling puppet across eqiad |
[production] |
| 23:05 |
<mutante> |
temp disabling puppet on all eqiad hosts via salt - during ganglia aggregator switch |
[production] |
| 22:51 |
<demon@tin> |
Synchronized php-1.29.0-wmf.7/extensions/Echo/includes/api: silence api warnings (duration: 02m 46s) |
[production] |
| 22:40 |
<demon@tin> |
Synchronized php-1.29.0-wmf.7/extensions/VisualEditor: silence some api warnings (duration: 02m 48s) |
[production] |
| 22:22 |
<demon@tin> |
Synchronized php-1.29.0-wmf.7/extensions/CodeReview/api/ApiQueryCodeComments.php: silence some warnings (duration: 02m 46s) |
[production] |
| 21:59 |
<mutante> |
rolling out exim4 upgrades (DSA 3747-1) on all remaning ones in codfw (all-codfw) |
[production] |
| 21:53 |
<ejegg> |
updated payments wiki from 21ea9bc4a46c96f84685b33ef06fd1b4c40e797e to 1f9ea809b3c32060222f4f6a70e725c433fe15b7 |
[production] |
| 21:53 |
<mutante> |
mx1001 - upgrading exim4 packages, exim4-daemon-heavy, forcing puppet run |
[production] |
| 21:35 |
<mutante> |
mx2001 - upgrading exim4 packages, daemon-heavey, forcing puppet run |
[production] |
| 21:22 |
<thcipriani@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.7 |
[production] |
| 20:58 |
<mutante> |
mendelevium (OTRS) - upgrade exim4 packages, force puppet run |
[production] |
| 20:45 |
<mutante> |
iridium (phabricator) - upgrade exim4 packages, force puppet run |
[production] |
| 20:29 |
<demon@tin> |
Synchronized docroot/wikipedia.org: removing junk 15 stuff (duration: 04m 50s) |
[production] |
| 20:18 |
<thcipriani@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.29.0-wmf.7 |
[production] |
| 20:07 |
<demon@tin> |
Synchronized docroot: Ok last docroot thing for today I promise (duration: 05m 00s) |
[production] |
| 19:52 |
<demon@tin> |
Synchronized docroot: Final bit of this round of docroot cleanup (duration: 05m 00s) |
[production] |
| 19:44 |
<thcipriani@tin> |
Finished scap: SemanticForms l10n cache rebuild for 1.29.0-wmf.7 (duration: 39m 21s) |
[production] |
| 19:35 |
<mutante> |
mira - upgrade exim, python-requests, linux-image |
[production] |
| 19:29 |
<mutante> |
rolled out exim4 upgrades (DSA 3747-1) on contint1001/2001, dbmonitor1001/2001, tungsten, seaborgium, hassium, labstore* |
[production] |
| 19:05 |
<thcipriani@tin> |
Started scap: SemanticForms l10n cache rebuild for 1.29.0-wmf.7 |
[production] |
| 18:34 |
<mutante> |
fermium (lists server) - upgrading exim packages, exim4-daemon-heavy, forcing puppet run |
[production] |
| 18:29 |
<yurik@tin> |
Finished deploy [graphoid/deploy@d20b00e]: (no message) (duration: 03m 11s) |
[production] |
| 18:26 |
<yurik@tin> |
Starting deploy [graphoid/deploy@d20b00e]: (no message) |
[production] |