|
2014-04-21
§
|
| 16:34 |
<ottomata> |
reinstalling elastic1013 (elastic1014 is still coming back online, but I don't want there to an extra eligible master for long) |
[production] |
| 16:04 |
<ottomata> |
reinstalling elastic1014 |
[production] |
| 15:22 |
<cmjohnson1> |
dataset2 going down to be relocated to the 12th floor |
[production] |
| 13:41 |
<manybubbles> |
rolling restart on some of the Elasticsearch servers to pick up new plugins. should not cause any trouble. |
[production] |
| 13:05 |
<Reedy> |
De-activated status.wm.o monitor for icinga due to false positive from HTTP auth |
[production] |
| 12:54 |
<paravoid> |
demoting myself, removing Commons crat/admin rights |
[production] |
| 12:41 |
<paravoid> |
escalating myself to Commons bureaucrat/admin, then adding GWToolset privileges |
[production] |
| 12:40 |
<paravoid> |
deleting 29 GWToolset XML under Swift's wikipedia-commons-gwtoolset-metadata container for user Fæ/ |
[production] |
| 11:51 |
<reedy> |
synchronized php-1.23wmf22/extensions/TimedMediaHandler 'I7483c8b7ec75f5149998da2b530ca04' |
[production] |
| 11:51 |
<paravoid> |
deactivating esams<->HE peering, >90% packet loss between lon<->nyc |
[production] |
| 11:49 |
<paravoid> |
deactivating eqiad<->HE peering, >90% packet loss between lon<->nyc |
[production] |
| 11:45 |
<reedy> |
synchronized php-1.23wmf22/extensions/TimedMediaHandler 'I7483c8b7ec75f5149998da2b530ca0467ac70de7' |
[production] |
| 03:55 |
<springle> |
reset pc100* slaves previously replicating from pmtpa |
[production] |
| 03:33 |
<ori> |
5.5k fatals over last 20 hrs, of which 3.5k are calls to doTransform() on a non-object at TimedMediaThumbnail.php:201, and 0.9k are Lua API OOMs at LuaSandbox/Engine.php:264 |
[production] |
| 03:30 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Apr 21 03:30:15 UTC 2014 (duration 30m 14s) |
[production] |
| 03:26 |
<ori> |
ap_busy_workers spike on image scalers eqiad, started ~2:55, subsided around ~3:20 |
[production] |
| 02:42 |
<LocalisationUpdate> |
completed (1.24wmf1) at 2014-04-21 02:42:30+00:00 |
[production] |
| 02:29 |
<LocalisationUpdate> |
completed (1.23wmf22) at 2014-04-21 02:29:49+00:00 |
[production] |
|
2014-04-18
§
|
| 21:00 |
<hashar> |
Jenkins renamed mw-jenkinsbot irc bot to wmf-insecte (french for "bug"). Updated IRC conf to point to chat.freenode.net:7000 with SSL. |
[production] |
| 19:02 |
<bblack> |
enabled cp30[14] varnish mobile frontends in esams pybal |
[production] |
| 17:50 |
<bblack> |
cp301[34] reinstalls complete, should stay ok in monitoring |
[production] |
| 17:48 |
<ottomata> |
resinsalling elastic1008 |
[production] |
| 16:20 |
<springle> |
db48 mysqld shutdown for decom |
[production] |
| 16:20 |
<bblack> |
ignore cp301[34] msgs, reinstalling them |
[production] |
| 16:10 |
<springle> |
db63 mysqld shutdown for decom |
[production] |
| 15:53 |
<ottomata> |
reinstalling elastic1007 |
[production] |
| 15:52 |
<springle> |
db48 mysqld set read_only, disabled m2 repl to db1048 |
[production] |
| 15:51 |
<ottomata> |
disabling puppet on elasti1007 and elastic1008 for reformatting |
[production] |
| 15:45 |
<mutante> |
DNS update - removing Tampa msbe/msfe |
[production] |
| 15:38 |
<Jeff_Green> |
switched mchenry to use db1048/db1049 for OTRS address lookups |
[production] |
| 15:24 |
<mutante> |
DNS update - removing all the Tampa mw/srv mgmt |
[production] |
| 15:15 |
<mutante> |
DNS update - removing lvs1-6 |
[production] |
| 14:54 |
<mutante> |
es5,es6 - revoke puppet certs, salt keys, icinga |
[production] |
| 14:51 |
<ottomata> |
powering down stat1 for decom |
[production] |
| 14:43 |
<mutante> |
ms-fe[14] - shutting down |
[production] |
| 14:41 |
<ottomata> |
disabling puppet on stat1 for decom |
[production] |
| 14:37 |
<mutante> |
ms-be 1-12, Tampa Swift boxes, shutdown |
[production] |
| 14:24 |
<mutante> |
ms-fe[14] - stop puppet,revoke certs,remove icinga |
[production] |
| 13:54 |
<mutante> |
ms-be1-12 - removing from puppet,salt,icinga |
[production] |
| 13:07 |
<mutante> |
Bugzilla Apache, changed SSL cipher suite in I7e9adc182dc ,might cost a a few % performance but zirconium had plenty |
[production] |
| 11:49 |
<hashar> |
removing mw-jenkinsbot (the wikimedia jenkins installation) from #wikimedia-labs |
[production] |
| 10:10 |
<hashar> |
Jenkins upgraded to 1.532.3. |
[production] |