|
2014-06-24
§
|
| 14:15 |
<hashar> |
Jenkins set SMTP server to wiki-mail.wikimedia.org smtp.pmtpa.wmnet got deleted |
[production] |
| 14:07 |
<hashar> |
Jenkins is back |
[production] |
| 13:59 |
<Krinkle> |
Build logs in Jenkins incorrectly render ansi color codes since it was upgraded to 0.4.0. Downgrading to 0.3.1 and restarting Jenkins. |
[production] |
| 09:55 |
<godog> |
removing old salt master cache on palladium, moved yesterday out of the way |
[production] |
| 06:59 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1049 (duration: 00m 08s) |
[production] |
| 06:24 |
<Nemo_bis> |
FYI no gerrit mail since yesterday 15 UTC, https://bugzilla.wikimedia.org/show_bug.cgi?id=67018 |
[production] |
| 02:48 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Tue Jun 24 02:47:14 UTC 2014 (duration 47m 13s) |
[production] |
| 02:26 |
<LocalisationUpdate> |
completed (1.24wmf10) at 2014-06-24 02:25:43+00:00 |
[production] |
| 02:14 |
<LocalisationUpdate> |
completed (1.24wmf9) at 2014-06-24 02:13:38+00:00 |
[production] |
|
2014-06-23
§
|
| 23:12 |
<maxsem> |
Synchronized php-1.24wmf9/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/#/c/141102/ (duration: 00m 06s) |
[production] |
| 23:12 |
<maxsem> |
Synchronized php-1.24wmf10/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/#/c/141102/ (duration: 00m 05s) |
[production] |
| 23:03 |
<maxsem> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/140897/ (duration: 00m 04s) |
[production] |
| 20:06 |
<subbu> |
deployed parsoid 392435a2 (deploy sha db94f88c) |
[production] |
| 19:22 |
<hashar> |
gallium / zuul : deleting /var/lib/zuul/git old Zuul repositories. They have been migrated to /srv/ssd/zuul/git/ ages ago |
[production] |
| 19:20 |
<jgage> |
ms-be3003 full root partition fixed, swift had written to /srv/swift-storage/sdk1 onto root due to umounted sdk1 |
[production] |
| 17:38 |
<bblack> |
lvs1005:eth3 was negotiated to 100mbps (???) - disable -> enable on switch fixed it |
[production] |
| 17:36 |
<godog> |
restarted salt-master on palladium, suspected job cleanup stuck |
[production] |
| 17:04 |
<bd808> |
Fixed dangling symlink for /etc/apache2/sites-enabled/logstash.wikimedia.org on logstash1001 by deleting symlink and forcing puppet run |
[production] |
| 16:49 |
<godog> |
added mw1149-52 back to pybal apache |
[production] |
| 16:33 |
<paravoid> |
switched inbound mail for all non-wikimedia.org domains from mchenry/sodium to polonium/mchenry (~16:00 + <= 1h TTL UTC) |
[production] |
| 15:13 |
<anomie> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Add a Library of Congress domain to wgCopyUploadsDomains [[gerrit:141308]] (duration: 00m 14s) |
[production] |
| 15:11 |
<anomie> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Adjust group rights on ruwiki [[gerrit:140910]] (duration: 00m 14s) |
[production] |
| 15:10 |
<anomie> |
Synchronized php-1.24wmf9/includes/api/ApiExpandTemplates.php: SWAT: Fix fatal in API action=expandtemplates with Scribunto [[gerrit:141416]] (duration: 00m 15s) |
[production] |
| 15:04 |
<anomie> |
Synchronized php-1.24wmf10/includes/api/ApiExpandTemplates.php: SWAT: Fix fatal in API action=expandtemplates with Scribunto [[gerrit:141417]] (duration: 00m 14s) |
[production] |
| 14:55 |
<andrewbogott> |
reenabling puppet on labstore1001, hoping it doesn't break labs |
[production] |
| 14:38 |
<hashar> |
Further upgraded Zuul up to upstream b8c24ce + our local hacks. Git tag is wmf-deploy-20140623-4 |
[production] |
| 14:14 |
<hashar> |
upgraded Zuul by one commit (that introduces swift supports though disabled it on our setup via a custom hack) |
[production] |
| 13:20 |
<paravoid> |
switching outbound email to polonium |
[production] |
| 12:17 |
<manybubbles> |
rebuilding Cirrus index on group0 wikis to pick up changes like results boosting from categories and wikitext search |
[production] |
| 10:37 |
<godog> |
powering down maerlant, decom-med |
[production] |
| 10:05 |
<godog> |
hardreset maerlant, stuck on console and no ssh |
[production] |
| 09:40 |
<paravoid> |
killing sodium's lighttpd compress cache |
[production] |
| 07:21 |
<_joe_> |
powercycled cp4018, stuck with a blank console |
[production] |
| 02:59 |
<springle> |
moving lighttpd compressed archives on sodium off / to regain inodes |
[production] |
| 02:46 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Jun 23 02:45:24 UTC 2014 (duration 45m 23s) |
[production] |
| 02:27 |
<LocalisationUpdate> |
completed (1.24wmf10) at 2014-06-23 02:25:53+00:00 |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.24wmf9) at 2014-06-23 02:13:53+00:00 |
[production] |
| 00:38 |
<legoktm> |
mail is stuck, lots of mails queued in exim |
[production] |