|
2014-07-10
§
|
| 09:07 |
<hashar> |
gallium err was July 5th and file was from a minute ago ... ignore me |
[production] |
| 09:06 |
<hashar> |
gallium deleted /var/lib/puppet/state/agent_catalog_run.lock from July 5th. Was preventing me to run <tt>puppet agent -tv</tt> |
[production] |
| 08:02 |
<oblivian> |
gracefulled all apaches |
[production] |
| 07:52 |
<_joe_> |
doing a tagged run of puppet on all appservers to sync apache config |
[production] |
| 06:40 |
<bblack> |
all normally-ulsfo traffic is back on ulsfo |
[production] |
| 05:53 |
<awight> |
edit CRM Drupal permissions |
[production] |
| 03:47 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Jul 10 03:46:36 UTC 2014 (duration 46m 35s) |
[production] |
| 03:12 |
<LocalisationUpdate> |
completed (1.24wmf12) at 2014-07-10 03:11:48+00:00 |
[production] |
| 02:49 |
<mutante> |
argon,netmon1001, graceful'led apaches |
[production] |
| 02:48 |
<mutante> |
netmon1001 - DocumentRoot [/etc/apache2/undef] does not exist |
[production] |
| 02:42 |
<LocalisationUpdate> |
completed (1.24wmf11) at 2014-07-10 02:41:29+00:00 |
[production] |
| 02:38 |
<mutante> |
argon,calcium,iron,rhenium,bast1001,oxygen,netmon1001 - upgraded SSL |
[production] |
| 01:47 |
<mutante> |
argon - Ignoring file 'puppet_base_2.7' in directory '/etc/apt/preferences.d/ |
[production] |
| 01:41 |
<awight> |
update crm schema to wmf_civicrm 7020 |
[production] |
| 01:40 |
<awight> |
update civicrm from 108802336e4d5f4aab9a6dbfa0ea434bddae0060 to 15cf86cb109a448f1982da9c91215eec73f28499 |
[production] |
| 01:38 |
<mutante> |
potassium,hydrogen,search1016,nitrogen,analytics1024,chromium - upgrade SSL |
[production] |
| 01:06 |
<bblack> |
cleared icinga downtimes for ulsfo (we now have some traffic back there) |
[production] |
| 00:50 |
<mattflaschen> |
Synchronized php-1.24wmf11/extensions/GuidedTour/: GuidedTour cherry-pick to 1.24wmf11 in support of GettingStarted anonymous editor acquisition test (duration: 00m 09s) |
[production] |
| 00:05 |
<maxsem> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/144938 (duration: 00m 04s) |
[production] |
|
2014-07-09
§
|
| 23:57 |
<maxsem> |
Finished scap: SWAT, GettingStarted introduced a new message (duration: 26m 31s) |
[production] |
| 23:44 |
<mutante> |
deleted systemusers group on neon & mw1077 (to check it doesnt break anything |
[production] |
| 23:31 |
<maxsem> |
Started scap: SWAT, GettingStarted introduced a new message |
[production] |
| 23:22 |
<maxsem> |
Synchronized php-1.24wmf11/extensions/GettingStarted/: (no message) (duration: 00m 03s) |
[production] |
| 23:17 |
<maxsem> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/144857/ (duration: 00m 04s) |
[production] |
| 22:42 |
<mark> |
Enabling PAIX BGP sessions on cr2-ulsfo |
[production] |
| 22:40 |
<mark> |
Enabling WMF HQ BGP sessions on cr1-ulsfo |
[production] |
| 22:38 |
<mark> |
Enabling TiNet transit links on cr1-ulsfo |
[production] |
| 22:35 |
<mark> |
Enabling WMF HQ BGP sessions on cr2-ulsfo |
[production] |
| 22:34 |
<mark> |
Enabling NTT and HE transit links on cr2-ulsfo |
[production] |
| 22:05 |
<mutante> |
restarted apache on zirconium for config change |
[production] |
| 20:07 |
<subbu> |
deployed parsoid 1632288d |
[production] |
| 18:36 |
<yurik> |
Synchronized php-1.24wmf12/extensions/: update to JsonConfig, ZeroBanner, ZeroPortal (duration: 01m 22s) |
[production] |
| 18:29 |
<yurik> |
Synchronized php-1.24wmf11/extensions/: update to JsonConfig, ZeroBanner, ZeroPortal (duration: 01m 24s) |
[production] |
| 17:17 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: eswiki cirrus (duration: 00m 04s) |
[production] |
| 16:51 |
<csteipp> |
Finished scap: Update CentralAuth for Global Rename (duration: 28m 46s) |
[production] |
| 16:22 |
<csteipp> |
Started scap: Update CentralAuth for Global Rename |
[production] |
| 16:17 |
<mark> |
ulsfo is now offline |
[production] |
| 16:17 |
<mark> |
Shutdown NTT BGP sessions on cr2-ulsfo |
[production] |
| 16:13 |
<mark> |
Shutdown TiNet BGP sessions on cr1-ulsfo |
[production] |
| 16:10 |
<mark> |
Shutdown IXP BGP sessions on cr2-ulsfo |
[production] |
| 16:10 |
<mark> |
Shutdown WMF HQ BGP sessions on cr2-ulsfo |
[production] |
| 16:09 |
<mark> |
Shutdown WMF HQ BGP sessions on cr1-ulsfo |
[production] |
| 16:03 |
<hoo> |
Synchronized php-1.24wmf12/extensions/Wikidata/: Update Wikibase to fix a fatal and various JS things (duration: 00m 14s) |
[production] |
| 14:13 |
<hashar> |
Jenkins: bringing back puppet-compiler02.eqiad.wmflabs node online. /tmp get filled when running huge catalog compilations which causes Jenkins to unpool the node :/ |
[production] |
| 13:30 |
<godog> |
reboot ms-be1005, raid controller confused (?) after disk replacement |
[production] |
| 12:52 |
<godog> |
umounted sdg1 on ms-be1005, device disappeared, errors in dmesg |
[production] |
| 12:36 |
<bblack> |
enabled amssq47 text frontend cache in pybal for esams |
[production] |
| 09:30 |
<hashar> |
restarted Zuul to clear out stalled items in queue |
[production] |
| 09:12 |
<hashar> |
Jenkins being slow because the mediawiki-core* jobs history cache has been wiped out while updating their configuration. Jenkins is busy processing the history :( |
[production] |
| 09:02 |
<hashar> |
Jenkins killing slave process on lanthanum. Some job is stalled and unrecoverable. |
[production] |