|
2015-01-13
ยง
|
| 22:52 |
<ori> |
Synchronized wmf-config/mc.php: Use UNIX domain socket for nutcracker on mw1030 & mw1031 (duration: 00m 05s) |
[production] |
| 22:37 |
<hashar> |
Restarted Zuul, deadlocked waiting for Gerrit |
[releng] |
| 22:37 |
<hashar> |
Restarted Zuul, deadlocked waiting for Gerrit |
[production] |
| 22:37 |
<ejegg> |
updated payments from c23cf16407ef200da446d81fb990abbe429fd378 to 46bd1611c56d6e31594d01aa345c35dd45dcf676 |
[production] |
| 21:38 |
<ori> |
deployment-prep upgraded nutcracker on mw1/mw2 to 0.4.0+dfsg-1+wm1 |
[releng] |
| 20:48 |
<Reedy> |
running mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki=mediawikiwiki --reindexAndRemoveOk --indexIdentifier=now |
[production] |
| 20:48 |
<reedy> |
Synchronized wmf-config/InitialiseSettings.php: mediawikiwiki content namespaces (duration: 00m 05s) |
[production] |
| 20:42 |
<reedy> |
Synchronized wmf-config/: Config updates (duration: 00m 06s) |
[production] |
| 20:41 |
<reedy> |
Synchronized database lists: wikidata dblist update (duration: 00m 06s) |
[production] |
| 19:38 |
<reedy> |
Synchronized wmf-config/InitialiseSettings.php: Add Renameuser debug log group (duration: 00m 09s) |
[production] |
| 19:24 |
<reedy> |
Synchronized wmf-config/Wikibase.php: bump cache epoch (duration: 00m 06s) |
[production] |
| 19:09 |
<reedy> |
Synchronized wmf-config/InitialiseSettings.php: monolog: enable for group0 + group1 wikis (duration: 00m 07s) |
[production] |
| 19:06 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Non Wikipedias to 1.25wmf14 |
[production] |
| 18:40 |
<ori> |
mw1062: sync-file failed, read-only file system. Host should be removed from dsh group. |
[production] |
| 18:38 |
<ori> |
Synchronized wmf-config/StartProfiler.php: xenon: Skip frames that don't have a 'phpStack' key (duration: 00m 06s) |
[production] |
| 17:43 |
<hashar> |
Restarted deadlocked Zuul , which drops ALL events. Reason is Gerrit lost connection with its database which is not handled by Zuul . See https://wikitech.wikimedia.org/wiki/Incident_documentation/20150106-Zuul |
[releng] |
| 17:39 |
<hashar> |
Zuul back in action. Got recheck or +2 again the changes that have been discarded. |
[production] |
| 17:37 |
<hashar> |
Restarting deadlocked Zuul , which drops ALL events. Reason is Gerrit lost connection with its database which is not handled by Zuul . See https://wikitech.wikimedia.org/wiki/Incident_documentation/20150106-Zuul |
[production] |
| 17:32 |
<James_F> |
No effect from restarting Gearman. Getting Timo to restart Zuul. |
[releng] |
| 17:30 |
<James_F> |
No effect. Restarting Gearman. |
[releng] |
| 17:27 |
<James_F> |
Trying a shutdown/re-enable of Jenkins. |
[releng] |
| 16:12 |
<demon> |
Synchronized wmf-config/: (no message) (duration: 00m 05s) |
[production] |
| 16:12 |
<demon> |
Synchronized flaggedrevs.dblist: (no message) (duration: 00m 05s) |
[production] |
| 16:02 |
<YuviPanda> |
restart mysql on virt1000, wikitech acting up again |
[production] |
| 15:23 |
<godog> |
upgrade and restart poolcounter on helium |
[production] |
| 15:17 |
<godog> |
upload poolcounter 1.0.3 to precise-wikimedia |
[production] |
| 15:17 |
<YuviPanda> |
restarted mysql on virt1000, wikitech failing with db errors. seems fine now (4 minutes ago) |
[production] |
| 15:17 |
<YuviPanda> |
test |
[production] |
| 15:14 |
<_joe_> |
jobrunners 100% on HHVM |
[production] |
| 15:14 |
<_joe_> |
reimaging mw1007 mw1008 |
[production] |
| 14:36 |
<bblack> |
ssl runtime config updated for +3DES/-RC4 ( I87616455abd58c986aa960348fc20c017f097716 ) |
[production] |
| 14:07 |
<_joe_> |
reimaging mw1005,mw1006 |
[production] |
| 13:59 |
<YuviPanda> |
running scap via jenkins, hitting buttons on https://integration.wikimedia.org/ci/job/beta-scap-eqiad/ |
[releng] |
| 13:58 |
<YuviPanda> |
scap failed |
[releng] |
| 13:58 |
<YuviPanda> |
running scap, because why not |
[releng] |
| 13:58 |
<YuviPanda> |
modified PrivateSettings.php to make it use wikiadmin user rather than mw user |
[releng] |
| 13:51 |
<YuviPanda> |
created user wikiadmin on deployment-db1 |
[releng] |
| 13:45 |
<Jeff_Green> |
package updates and reboots on many fundraising hosts |
[production] |
| 11:30 |
<_joe_> |
reimaging mw1003, mw1004 |
[production] |
| 11:30 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1067, warm up (duration: 00m 05s) |
[production] |
| 11:20 |
<springle> |
upgrade db1067 trusty |
[production] |
| 10:30 |
<hoo> |
Set email for global account "Liberipedia" as per https://phabricator.wikimedia.org/T76321 |
[production] |
| 10:19 |
<_joe_> |
reimaging mw1001, mw1002 |
[production] |
| 10:02 |
<_joe_> |
net.ipv4.tcp_tw_reuse = 1 on mw1223 |
[production] |
| 08:58 |
<_joe_> |
raising the net.ipv4.ip_local_port_range on mw1196 |
[production] |
| 08:43 |
<_joe_> |
raising the net.ipv4.ip_local_port_range on mw1230 |
[production] |
| 07:30 |
<springle> |
ongoing schema changes T86415 externallinks, codfw first |
[production] |
| 04:55 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1067 (duration: 00m 05s) |
[production] |
| 04:31 |
<James_F> |
Zuul now appears fixed. |
[releng] |
| 04:29 |
<marktraceur> |
FORCE RESTART ZUUL (James_F told me to) |
[releng] |