|
2014-07-31
§
|
| 10:49 |
<hashar> |
Jenkins: attempting to poll a Trusty slave (integration-slave1004-trusty [10.68.17.148] with label <tt>UbuntuTrusty</tt>). |
[production] |
| 10:32 |
<hashar> |
Jenkins: tweaking jobs labels, that might eventually screw up Zuul/Jenkins entirely. |
[production] |
| 08:43 |
<_joe_> |
start rolling reload of nginx to catch up with the new ssl config |
[production] |
| 06:50 |
<springle> |
labsdb1001 migration complete, should be all systems go |
[production] |
| 03:19 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Jul 31 03:18:07 UTC 2014 (duration 18m 6s) |
[production] |
| 02:36 |
<LocalisationUpdate> |
completed (1.24wmf15) at 2014-07-31 02:35:29+00:00 |
[production] |
| 02:20 |
<LocalisationUpdate> |
completed (1.24wmf14) at 2014-07-31 02:19:17+00:00 |
[production] |
| 02:06 |
<springle> |
labsdb1001 migrating to mariadb 10, expect read-only and downtime, see labs-l |
[production] |
|
2014-07-30
§
|
| 23:27 |
<maxsem> |
Synchronized php-1.24wmf15/extensions/MwEmbedSupport/: (no message) (duration: 00m 03s) |
[production] |
| 23:27 |
<maxsem> |
Synchronized php-1.24wmf15/extensions/Wikidata/: (no message) (duration: 00m 08s) |
[production] |
| 23:27 |
<maxsem> |
Synchronized php-1.24wmf15/extensions/SyntaxHighlight_GeSHi/: (no message) (duration: 00m 05s) |
[production] |
| 23:24 |
<maxsem> |
Synchronized php-1.24wmf14/extensions/Wikidata: (no message) (duration: 00m 11s) |
[production] |
| 23:13 |
<maxsem> |
Synchronized wmf-config: (no message) (duration: 00m 05s) |
[production] |
| 21:04 |
<AaronSchulz> |
Started populateBacklinkNamespace.php on wikidata and commons |
[production] |
| 21:02 |
<bblack> |
turned icinga email/sms back on |
[production] |
| 20:24 |
<bblack> |
icinga back online again |
[production] |
| 19:58 |
<bblack> |
shutting off icinga to make some optimizations |
[production] |
| 19:20 |
<bblack> |
icinga is now substantially back online. email/sms still disabled for now, and downtimes/acks need to be re-added for known issues |
[production] |
| 19:06 |
<csteipp> |
Synchronized php-1.24wmf14/includes/: (no message) (duration: 00m 05s) |
[production] |
| 19:04 |
<csteipp> |
Synchronized php-1.24wmf15/includes/: (no message) (duration: 00m 07s) |
[production] |
| 18:59 |
<bblack> |
icinga coming back up again for the first time, expect random strangeness to be ignored |
[production] |
| 18:46 |
<bblack> |
temporarily hard-disabling email/sms from icinga via 'mv /usr/bin/mail /usr/bin/mail-disabled' on neon to prevent icinga spam on next startup attempt |
[production] |
| 17:55 |
<bblack> |
stopping icinga service for now while working out other details |
[production] |
| 17:25 |
<tacotuesday> |
repooled elastic1018 and elastic1019 as well |
[production] |
| 17:21 |
<Coren> |
labmon1001 rebooting (final check for proper raid+lvm autodetection) |
[production] |
| 17:08 |
<bblack> |
working on bringing up new neon install (first puppet run, etc) |
[production] |
| 17:01 |
<Coren> |
labmon1001 rebooting (partitioning changes on primary disks) |
[production] |
| 16:53 |
<tacotuesday> |
elastic1017 repooled, shards allocating |
[production] |
| 16:13 |
<bd808> |
scap and dologmsg from tin won't work until neon is back up and running tcpircbot |
[production] |
| 16:07 |
<bd808|deploy> |
Synchronized touch: no-op sync to test scap update (duration: 00m 05s) |
[production] |
| 16:06 |
<bd808|deploy> |
scap announce failed -- timeout connecting to tcpircbot on neon.wikimedia.org |
[production] |
| 16:04 |
<bd808|deploy> |
Updated scap to 4871208 (rely on $PATH for scap scripts) |
[production] |
| 15:21 |
<hoo> |
Synchronized php-1.24wmf15/extensions/Wikidata/extensions/Wikibase/lib/resources/wikibase.js: touch (duration: 00m 20s) |
[production] |
| 15:17 |
<hashar> |
upgrading php5 on jenkins slaves |
[production] |
| 15:07 |
<cmjohnson1> |
shutting down neon |
[production] |
| 14:46 |
<demon> |
Synchronized wmf-config/CirrusSearch-production.php: (no message) (duration: 00m 04s) |
[production] |
| 14:35 |
<demon> |
Synchronized wmf-config/PrivateSettings.php: Swift config for Cirrus (duration: 00m 08s) |
[production] |
| 14:30 |
<godog> |
rolling restart of ms-fe* to pick up search backup user |
[production] |
| 14:17 |
<bblack> |
rebooting neon again, trying to fix the disk situation |
[production] |
| 14:11 |
<Coren> |
reinstalling labmon1001 -> change disk partitioning scheme |
[production] |
| 13:50 |
<springle> |
neon read-only fs. fsck + reboot |
[production] |
| 13:17 |
<manybubbles> |
rebuiding Cirrus index for commons to pick up weighted all field |
[production] |
| 11:17 |
<_joe_> |
enabling puppet on all mw* servers |
[production] |
| 11:15 |
<_joe_> |
re-enabling puppet on mw1019, last bunch of tests, then re-enabling globally |
[production] |
| 10:58 |
<_joe_> |
re-enabling puppet on mw1018, testwiki upgraded to the new config and looks fine |
[production] |
| 09:25 |
<godog> |
set weight for ms-be1014 and ms-be1015 to 2300 |
[production] |
| 08:58 |
<_joe_> |
stopping puppet on the appservers, in preparation for releasing change 148099 |
[production] |
| 08:30 |
<_joe_> |
powercycling neon, doesn't respond to requests, ssh hangs, console dark |
[production] |
| 06:41 |
<springle> |
labsdb1001 work in progress; it may misbehave. see labs-l for updates |
[production] |
| 04:29 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Jul 30 04:27:56 UTC 2014 (duration 27m 55s) |
[production] |