|
2014-07-09
§
|
| 16:09 |
<mark> |
Shutdown WMF HQ BGP sessions on cr1-ulsfo |
[production] |
| 16:03 |
<hoo> |
Synchronized php-1.24wmf12/extensions/Wikidata/: Update Wikibase to fix a fatal and various JS things (duration: 00m 14s) |
[production] |
| 14:13 |
<hashar> |
Jenkins: bringing back puppet-compiler02.eqiad.wmflabs node online. /tmp get filled when running huge catalog compilations which causes Jenkins to unpool the node :/ |
[production] |
| 13:30 |
<godog> |
reboot ms-be1005, raid controller confused (?) after disk replacement |
[production] |
| 12:52 |
<godog> |
umounted sdg1 on ms-be1005, device disappeared, errors in dmesg |
[production] |
| 12:36 |
<bblack> |
enabled amssq47 text frontend cache in pybal for esams |
[production] |
| 09:30 |
<hashar> |
restarted Zuul to clear out stalled items in queue |
[production] |
| 09:12 |
<hashar> |
Jenkins being slow because the mediawiki-core* jobs history cache has been wiped out while updating their configuration. Jenkins is busy processing the history :( |
[production] |
| 09:02 |
<hashar> |
Jenkins killing slave process on lanthanum. Some job is stalled and unrecoverable. |
[production] |
| 08:53 |
<godog> |
upgrade ms-be1013/1014/1015 (zone5) to icehouse swift |
[production] |
| 08:51 |
<hashar> |
Jenkins migrating jobs to use $ZUUL_URL instead of git://zuul.eqiad.wmnet Preparing to scale out Zuul merger to several nodes |
[production] |
| 08:19 |
<godog> |
upgrade ms-be1009/1010/1011 (zone4) to swift icehouse |
[production] |
| 08:04 |
<hashar> |
Jenkins: granted matanya the ability to manually trigger builds. Use case: the puppet compiler! |
[production] |
| 08:02 |
<godog> |
upgrade ms-be1005/1006/1007 (zone3) to swift icehouse |
[production] |
| 03:37 |
<mutante> |
ran puppet on neon - false puppet failure alarms |
[production] |
| 02:55 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Jul 9 02:54:37 UTC 2014 (duration 54m 36s) |
[production] |
| 02:26 |
<LocalisationUpdate> |
completed (1.24wmf12) at 2014-07-09 02:25:33+00:00 |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.24wmf11) at 2014-07-09 02:14:38+00:00 |
[production] |
| 01:26 |
<mutante> |
Bugzilla - enabled https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security |
[production] |
| 00:50 |
<mutante> |
restarted gitblit service |
[production] |
|
2014-07-08
§
|
| 22:50 |
<mutante> |
radon (phab)- package and kernel upgrades, rebooting |
[production] |
| 20:22 |
<legoktm> |
finished running migrateAccount.php --attachbroken --attachmissing (bug 61876) |
[production] |
| 20:07 |
<legoktm> |
finished migrateAccount.php --safe, now starting migrateAccount.php --attachbroken |
[production] |
| 20:05 |
<mutante> |
restarted apache on ytterbium |
[production] |
| 19:47 |
<K4-713> |
updated payments fraud filters again |
[production] |
| 19:47 |
<legoktm> |
running migrateAccount.php --safe for accounts only existing on one wiki (bug 39817) |
[production] |
| 19:27 |
<mutante> |
this should have fixed all the services behind misc. varnish now getting an actual "A" rating on ssllabs |
[production] |
| 19:21 |
<mutante> |
arr, i meant "nginx", not varnish |
[production] |
| 19:15 |
<mutante> |
restarting varnish on cp1043/cp1044 (misc cluster) |
[production] |
| 18:56 |
<cmjohnson1> |
disconnecting serial cable from psw1-c2-eqiad |
[production] |
| 18:50 |
<csteipp> |
patch for bug66608 deployed to wmf11/12 |
[production] |
| 18:50 |
<K4-713> |
updated fraud filters on payments cluster |
[production] |
| 18:28 |
<reedy> |
Synchronized robots-private.txt: (no message) (duration: 00m 14s) |
[production] |
| 18:27 |
<reedy> |
Synchronized wmf-config/: (no message) (duration: 00m 15s) |
[production] |
| 18:20 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Non Wikipedias to 1.24wmf12 |
[production] |
| 15:22 |
<reedy> |
Purged l10n cache for 1.24wmf10 |
[production] |
| 15:21 |
<reedy> |
Purged l10n cache for 1.24wmf9 |
[production] |
| 15:15 |
<anomie> |
Synchronized php-1.24wmf11/extensions/Scribunto/: SWAT: Fix regression in os.date and os.time at module scope [[gerrit:144559]] (duration: 00m 10s) |
[production] |
| 15:14 |
<anomie> |
Synchronized php-1.24wmf12/extensions/Scribunto/: SWAT: Fix regression in os.date and os.time at module scope [[gerrit:144511]] (duration: 00m 11s) |
[production] |
| 15:10 |
<anomie> |
Synchronized php-1.24wmf11/extensions/UploadWizard/UploadWizard.config.php: SWAT: Flickr API is https-only now [[gerrit:144584]] (duration: 00m 10s) |
[production] |
| 15:04 |
<anomie> |
Synchronized php-1.24wmf12/extensions/UploadWizard/UploadWizard.config.php: SWAT: Flickr API is https-only now [[gerrit:144583]] (duration: 00m 10s) |
[production] |
| 13:34 |
<springle> |
slow transaction rollback in progress on db1001 librenms. other databases not affected, but librenms writes are timing out |
[production] |
| 13:32 |
<cmjohnson1> |
replacing disk disk 6 ms-be1005 |
[production] |
| 13:30 |
<cmjohnson1> |
replacing disk 4 ms-be1007 |
[production] |
| 12:38 |
<YuviPanda> |
disregard previous log message, was meant for labs |
[production] |
| 12:37 |
<YuviPanda> |
graphite reduced metrics count from 65k to 25k, monitoring io performance |
[production] |
| 06:57 |
<springle> |
Synchronized wmf-config/db-eqiad.php: raise db traffic samplers to normal load (duration: 00m 06s) |
[production] |
| 05:10 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1010, warm up (duration: 00m 06s) |
[production] |
| 04:51 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1010 for upgrade (duration: 00m 06s) |
[production] |
| 03:27 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Tue Jul 8 03:25:51 UTC 2014 (duration 25m 50s) |
[production] |