|
2014-09-30
§
|
| 21:23 |
<Nemo_bis> |
widespread reproducible 503 errors on wikidata and elsewhere |
[production] |
| 20:55 |
<andrewbogott> |
powering down virt0, just to see what breaks |
[production] |
| 20:48 |
<andrewbogott> |
shutting down pdns on virt0 |
[production] |
| 20:48 |
<andrewbogott> |
shutting down opendj on virt (temporary, a preview of tomorrow) |
[production] |
| 18:50 |
<reedy> |
Synchronized wmf-config/: (no message) (duration: 00m 16s) |
[production] |
| 18:49 |
<reedy> |
Synchronized database lists: (no message) (duration: 00m 15s) |
[production] |
| 18:41 |
<mutante> |
pc1001-1003 - can't generate tmp files for percona monitoring checks -> puppet fail |
[production] |
| 18:24 |
<mutante> |
killing silver from icinga and puppet |
[production] |
| 18:23 |
<reedy> |
Synchronized wmf-config/: (no message) (duration: 00m 16s) |
[production] |
| 18:21 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.25wmf1 |
[production] |
| 18:05 |
<ori> |
Synchronized wmf-config/HHVMRequestInit.php: (no message) (duration: 00m 07s) |
[production] |
| 18:04 |
<K4-713> |
re-enabled all queue consumers |
[production] |
| 17:56 |
<ejegg> |
updated civicrm from e83c999f39e6ae847d9b48e38c8c825fc10d1635 to b6c350f620c8dc1f3410de179c19cbcbdeb62270 |
[production] |
| 17:19 |
<K4-713> |
disabled qc jobs and TY mail send for pending civi deploy |
[production] |
| 15:45 |
<hashar> |
Updating our Jenkins job builder fork 686265a..ee80dbc (no job changed) |
[production] |
| 15:42 |
<bblack> |
rebooting mexia |
[production] |
| 15:33 |
<demon> |
Synchronized docroot/bits/favicon/wikipedia.ico: Favicons are my favorite icons, especially when they're only 18% of the size of the original (duration: 00m 04s) |
[production] |
| 15:16 |
<demon> |
Synchronized php-1.25wmf1/extensions/Wikidata: (no message) (duration: 00m 11s) |
[production] |
| 15:14 |
<demon> |
Synchronized php-1.25wmf1/extensions/VisualEditor: (no message) (duration: 00m 08s) |
[production] |
| 15:12 |
<akosiaris> |
merging https://gerrit.wikimedia.org/r/#/c/163735/1, changing the LDAP master from sanger to ldap-mirror for inbound mail |
[production] |
| 15:12 |
<andrewbogott> |
running sync-common on virt1000 |
[production] |
| 15:12 |
<demon> |
Synchronized visualeditor.dblist: (no message) (duration: 00m 04s) |
[production] |
| 15:12 |
<demon> |
Synchronized visualeditor-default.dblist: (no message) (duration: 00m 04s) |
[production] |
| 15:06 |
<demon> |
Synchronized wmf-config/Wikibase.php: (no message) (duration: 00m 04s) |
[production] |
| 15:06 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 06s) |
[production] |
| 14:26 |
<_joe_> |
restarted apache on mw1196, lots of apc errors |
[production] |
| 14:22 |
<oblivian> |
gracefulled all apaches |
[production] |
| 12:10 |
<mark> |
Stopped exim daemon on mchenry |
[production] |
| 09:41 |
<godog> |
removed obsolete /etc/puppet/hiera from strontium and palladium, /etc/puppet/hieradata is the new location |
[production] |
| 09:24 |
<godog> |
reboot ms-be2001 as a test |
[production] |
| 04:18 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Tue Sep 30 04:18:48 UTC 2014 (duration 18m 47s) |
[production] |
| 03:17 |
<LocalisationUpdate> |
completed (1.25wmf1) at 2014-09-30 03:17:15+00:00 |
[production] |
| 02:41 |
<ori> |
Synchronized 503.html: Ia88b306ef: Make the 503 error page consistent with other 5xx error pages (duration: 00m 08s) |
[production] |
| 02:34 |
<LocalisationUpdate> |
completed (1.24wmf22) at 2014-09-30 02:34:07+00:00 |
[production] |
| 01:00 |
<Krinkle> |
Jenkins connection seemed in order with integration-slave1007 and 8, but disconnecting and relaunching the slave agents immediately resulted in them getting jobs assigned. Cause unknown, problem resolved for now. |
[production] |
| 00:58 |
<Krinkle> |
integration-slave1007 and integration-slave1008 have not gotten any jobs in the past 24h. integration-slave1006 however has gotten loads of action. Investigating load balancing issue. |
[production] |
| 00:24 |
<mutante> |
linne - shutting down, revoking puppet cert, salt key, puppet/icinga ... |
[production] |
| 00:12 |
<maxsem> |
Synchronized w/skins-1.5: (no message) (duration: 00m 03s) |
[production] |
| 00:12 |
<MaxSem> |
https://gerrit.wikimedia.org/r/#/c/162520/ broke stuff, reverted |
[production] |
| 00:10 |
<maxsem> |
Synchronized live-1.5: (no message) (duration: 00m 03s) |
[production] |
|
2014-09-29
§
|
| 23:59 |
<maxsem> |
Synchronized w: https://gerrit.wikimedia.org/r/#/c/162520/ (duration: 00m 03s) |
[production] |
| 23:58 |
<maxsem> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/163773/ (duration: 00m 03s) |
[production] |
| 23:50 |
<maxsem> |
Synchronized php-1.25wmf1/extensions/MultimediaViewer/: second try... (duration: 00m 04s) |
[production] |
| 23:42 |
<maxsem> |
Finished scap: SWATting a bunch of stuff (duration: 18m 44s) |
[production] |
| 23:32 |
<andrewbogott> |
stopped apache, nova-scheduler, keystone, puppetmaster on virt0 |
[production] |
| 23:31 |
<bd808> |
/var/lib/jenkins-slave/tmpfs 100% full on lanthanum.eqiad.wmnet |
[production] |
| 23:26 |
<andrewbogott> |
disabling puppet on virt0 so I can kill off services one by one... |
[production] |
| 23:23 |
<maxsem> |
Started scap: SWATting a bunch of stuff |
[production] |
| 23:17 |
<maxsem> |
Synchronized docroot/: <mutante> he killed the dolphin (duration: 00m 06s) |
[production] |
| 23:13 |
<Reedy> |
dist-upgraded logstash1001 and reboot |
[production] |