|
2016-10-26
§
|
| 10:13 |
<bblack> |
rebooting ulsfo lvs primaries (lvs400[12]) |
[production] |
| 10:02 |
<bblack> |
rebooting codfw lvs primaries (lvs200[1-3]) |
[production] |
| 09:52 |
<jynus> |
starting schema change (imagelinks) on s1 T139090 |
[production] |
| 08:43 |
<elukey> |
increasing the AQS cassandra system_auth keyspace replication from 1 to 6 (and running nodetool-{a,b} repair system_auth on all nodes) |
[production] |
| 08:29 |
<elukey> |
downgraded memcached on mc2009 to the Debian Jessie version (was part of a performance experiment) |
[production] |
| 08:25 |
<dcausse> |
elastic@eqiad reindexing enwiki (take 3) with BM25 from wasat.codfw.wmnet T147508 (logs in ~dcausse/bm25_reindex/cirrus_log) |
[production] |
| 08:06 |
<marostegui> |
Stoppping replication on db2058 - using it to clone another host - T146261 |
[production] |
| 07:47 |
<moritzm> |
bounced ntp on oxygen (stuck in XFAC state) |
[production] |
| 07:45 |
<moritzm> |
rebooting mc* servers in codfw for kernel update |
[production] |
| 07:20 |
<moritzm> |
rebooting oxygen for kernel update |
[production] |
| 07:14 |
<marostegui> |
Deploying ALTER table s4 commonswiki.templatelinks - db2051 - T149079 |
[production] |
| 07:10 |
<moritzm> |
rebooting nescio for kernel update |
[production] |
| 06:34 |
<moritzm> |
repooled mw2098 (was previously down for hardware check) |
[production] |
| 03:00 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Wed Oct 26 03:00:28 UTC 2016 (duration 5m 28s) |
[production] |
| 02:55 |
<l10nupdate@tin> |
scap sync-l10n completed (1.28.0-wmf.23) (duration: 10m 13s) |
[production] |
| 02:28 |
<l10nupdate@tin> |
scap sync-l10n completed (1.28.0-wmf.22) (duration: 09m 38s) |
[production] |
| 01:36 |
<mutante> |
lead - (formerly gerrit) - shutdown -h now (T147905) |
[production] |
| 01:13 |
<mutante> |
palladium - shutdown -h now |
[production] |
| 00:10 |
<mutante> |
restarted icinga-wm, now there is /var/log/icinga/irc.log, it should talk now, but doesnt |
[production] |
| 00:04 |
<mutante> |
let icinga own /var/log/icinga on einsteinium, restart icinga |
[production] |
|
2016-10-25
§
|
| 23:44 |
<addshore@mira> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:293243]] Add a project namespace on tg.wikipedia (duration: 00m 47s) |
[production] |
| 23:41 |
<addshore@mira> |
Synchronized wmf-config/CommonSettings.php: [[gerrit:315121]] Stop adding Category:Uploaded_with_UploadWizard (duration: 00m 47s) |
[production] |
| 23:39 |
<addshore@mira> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:318013]] Enable static maps on testwiki (duration: 00m 48s) |
[production] |
| 23:26 |
<mutante> |
Submitted 'deactivate node' for palladium.eqiad.wmnet |
[production] |
| 23:21 |
<mutante> |
removed palladium from puppet (T147320). puppet node clean |
[production] |
| 23:02 |
<maxsem@mira> |
Synchronized php-1.28.0-wmf.23/extensions/ZeroBanner/: https://gerrit.wikimedia.org/r/#/c/318004/ (duration: 00m 50s) |
[production] |
| 23:00 |
<maxsem@mira> |
Synchronized php-1.28.0-wmf.23/extensions/ZeroPortal/: https://gerrit.wikimedia.org/r/#/c/318004/ (duration: 01m 32s) |
[production] |
| 22:49 |
<ejegg> |
updated CiviCRM from 844495a64a34f16167a5f5300826b9531ae360dd to 586433b81decb90b80c40acdbfde4548ca67af43 |
[production] |
| 22:34 |
<Pchelolo> |
revert RESTBase to f9017adc |
[production] |
| 22:33 |
<ejegg> |
updated payments-wiki from 1e8f6a22b71710163fc4e4d90ea03a9729ab33d6 to df4c72dbd328307019a82449527dd13c84bf41ab |
[production] |
| 22:02 |
<Pchelolo> |
RESTBase update to 3e53f00e |
[production] |
| 21:55 |
<Pchelolo> |
RESTBase update to 3e53f00e - staging |
[production] |
| 21:54 |
<ejegg> |
updated SmashPig from 31c0757ca17499deac6a8f49eb7ed17924630419 to daba8c07108246261ca7ec557e3f1e5da430826e |
[production] |
| 21:51 |
<maxsem@mira> |
Synchronized php-1.28.0-wmf.23/extensions/Graph/: https://gerrit.wikimedia.org/r/#/c/317989/ (duration: 01m 23s) |
[production] |
| 20:49 |
<ejegg> |
updated civicrm from 9e288691de2e918d00928c9164e0f2389563888d to 844495a64a34f16167a5f5300826b9531ae360dd |
[production] |
| 20:42 |
<maxsem@mira> |
Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/#/c/317879/ (duration: 00m 47s) |
[production] |
| 20:32 |
<ostriches> |
gerrit doing a quick reboot, config pick up |
[production] |
| 20:16 |
<filippo@mira> |
Synchronized wmf-config/ProductionServices.php: Put back potassium as poolcounter1002 (duration: 00m 51s) |
[production] |
| 20:14 |
<demon@mira> |
Synchronized php-1.28.0-wmf.23/extensions/CiteThisPage/SpecialCiteThisPage.php: T149112 (duration: 01m 39s) |
[production] |
| 19:41 |
<dcausse> |
elastic@eqiad reindexing enwiki with BM25 from terbium T147508 (logs in ~dcausse/bm25_reindex/cirrus_log) |
[production] |
| 19:23 |
<hasharAway> |
Python PyPi mirror has some issue. Impacts all CI jobs relying on tox https://status.python.org/ |
[production] |
| 19:22 |
<twentyafterfour> |
phabricator is back from reboot and it appears that all is well |
[production] |
| 19:19 |
<twentyafterfour> |
twentyafterfour@iridium: The system is going down for reboot NOW! |
[production] |
| 19:11 |
<twentyafterfour> |
rebooting iridium (phabricator) in ~ 3 minutes |
[production] |
| 19:11 |
<demon@mira> |
rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.23 |
[production] |
| 19:03 |
<ebernhardson@mira> |
Synchronized php-1.28.0-wmf.22/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: (no message) (duration: 00m 47s) |
[production] |
| 16:08 |
<gehel> |
restarting ferm on elastic2020 |
[production] |
| 16:04 |
<gehel> |
delete dangling indices on elasticsearch codfw: jawiki_general_first, jawiki_content_first, zhwiki_general_first and zhwiki_content_first |
[production] |
| 15:21 |
<ori> |
Synchronized wmf-config/throttle.php: I049bd463: Use correct IP for Vanderbilt 2016-10-25 edit-a-thon throttle exception (T149063) (duration: 01m 20s) |
[production] |
| 14:32 |
<gehel> |
reboot of wdqs cluster eqiad for kernel upgrade |
[production] |