|
2017-06-15
§
|
| 18:58 |
<ladsgroup@tin> |
Started deploy [ores/deploy@ab88a74]: Deploying gerrit:359224/1 for missing config variables |
[production] |
| 18:44 |
<paravoid> |
restarting all puppetmasters |
[production] |
| 18:40 |
<paravoid> |
temporarily stopping icinga-wm |
[production] |
| 18:27 |
<demon@tin> |
Synchronized wmf-config/CirrusSearch-common.php: Remove quirks and enable token_count_router thingie (duration: 00m 44s) |
[production] |
| 18:16 |
<demon@tin> |
Synchronized php-1.30.0-wmf.5/includes/libs/objectcache/MultiWriteBagOStuff.php: T167465 (duration: 00m 44s) |
[production] |
| 18:14 |
<demon@tin> |
Synchronized wmf-config/InitialiseSettings.php: T167617 (duration: 00m 44s) |
[production] |
| 18:12 |
<demon@tin> |
Synchronized wmf-config/FeaturedFeedsWMF.php: T167617 (duration: 00m 44s) |
[production] |
| 17:50 |
<mutante> |
install2002 - re-enabled puppet, reverted live hack, back to normal (issue seems to be NIC or other) |
[production] |
| 17:28 |
<mutante> |
install2002 - temp disabling puppet and applying hot fix to debug install issue for papaul |
[production] |
| 17:27 |
<bblack> |
disabling puppet on cp*wmnet to avoid puppet races on https://gerrit.wikimedia.org/r/#/c/341729 merge |
[production] |
| 14:39 |
<gehel> |
killing stuck replication on maps1001 |
[production] |
| 14:38 |
<krinkle@tin> |
Synchronized wmf-config/CommonSettings.php: no-op Ifc7b1ea80 - Remove EtcdConfig from beta (duration: 00m 45s) |
[production] |
| 13:24 |
<gehel> |
elasticsearch upgrade to 5.3.2 on relforge cluster completed, cluster still recovering - T163708 |
[production] |
| 13:23 |
<aude@tin> |
Synchronized wmf-config/Wikibase.php: Add constraints statements section on Wikidata T167126 (duration: 00m 43s) |
[production] |
| 13:19 |
<dcausse> |
[cirrus] reindexing all zh wikis (eqiad & codfw) |
[production] |
| 13:14 |
<aude@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable BM25 for Chinese wikis (duration: 00m 44s) |
[production] |
| 13:13 |
<aude@tin> |
Synchronized tests/cirrusTest.php: (no justification provided) (duration: 00m 45s) |
[production] |
| 13:02 |
<gehel> |
starting elasticsearch upgrade to 5.3.2 on relforge cluster - T163708 |
[production] |
| 12:14 |
<gehel> |
restart elasticsearch on relforge1001 to validate latest config changes |
[production] |
| 10:16 |
<moritzm> |
rollout remaining systemd updates from jessie point release |
[production] |
| 09:14 |
<jynus> |
shutting down and deleting data at pc1004 for cloning from db1096 |
[production] |
| 09:10 |
<hashar> |
Jenkins back up and happy. |
[production] |
| 09:05 |
<moritzm> |
reenable puppet on notebook1002, was disabled for the merge of the zookeeper role refactor two days ago, can be re-enabled now |
[production] |
| 09:04 |
<hashar> |
Restarting Jenkins. It seems I managed to deadlock it |
[production] |
| 08:52 |
<ariel@tin> |
Finished deploy [dumps/dumps@1734c6d]: history dump rebalance script, fixup for extension script dumps, root logger for misc dumps (duration: 00m 02s) |
[production] |
| 08:52 |
<ariel@tin> |
Started deploy [dumps/dumps@1734c6d]: history dump rebalance script, fixup for extension script dumps, root logger for misc dumps |
[production] |
| 08:40 |
<gehel> |
restart relforge1001 to validate latest config changes |
[production] |
| 08:16 |
<akosiaris@tin> |
Finished deploy [citoid/deploy@ba0db9c]: Remove the bad PMCID test from spec (duration: 07m 44s) |
[production] |
| 08:09 |
<akosiaris@tin> |
Started deploy [citoid/deploy@ba0db9c]: Remove the bad PMCID test from spec |
[production] |
| 08:02 |
<moritzm> |
updating HHVM on terbium/wasat to 3.18 |
[production] |
| 07:57 |
<akosiaris@tin> |
Finished deploy [citoid/deploy@ba0db9c]: Remove the bad PMCID test from spec (duration: 00m 38s) |
[production] |
| 07:56 |
<akosiaris@tin> |
Started deploy [citoid/deploy@ba0db9c]: Remove the bad PMCID test from spec |
[production] |
| 07:48 |
<akosiaris> |
schedule 2 hours downtime for all citoid endpoints health on scb boxes |
[production] |
| 06:08 |
<marostegui> |
Deploy alter table s2 - labsdb1003 - T166205 |
[production] |
| 05:50 |
<marostegui> |
Deploy alter table s2 - db1018 - T166205 |
[production] |
| 05:49 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Add comments to db1018 current status - T166205 (duration: 00m 43s) |
[production] |
| 05:41 |
<marostegui> |
Deploy alter table s4 - dbstore1001 - T166206 |
[production] |
| 05:22 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1036 - T166205 (duration: 00m 44s) |
[production] |
| 02:50 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Thu Jun 15 02:50:16 UTC 2017 (duration 6m 48s) |
[production] |
| 02:43 |
<l10nupdate@tin> |
scap sync-l10n completed (1.30.0-wmf.5) (duration: 07m 34s) |
[production] |
| 02:26 |
<l10nupdate@tin> |
scap sync-l10n completed (1.30.0-wmf.4) (duration: 09m 15s) |
[production] |
| 01:17 |
<mutante> |
releases1001 - reinstalling with stretch |
[production] |
| 00:15 |
<mutante> |
dumpsdata1001 - was reported in icinga as CRIT systemdstate - reason was puppet service was failed with "Invalid value '"no"' for boolean parameter: daemonize" (it was ok on other hosts??). commented the option, stopped puppet, systemctl reset-failed - which made it recover (T165368) |
[production] |
| 00:02 |
<twentyafterfour> |
Deploying phabricator update (tagged release/2017-06-14/1) details: https://phabricator.wikimedia.org/project/view/2831/ |
[production] |
|
2017-06-14
§
|
| 23:55 |
<mutante> |
mwreleases: revoke puppet cert, delete salt key, remove from icinga. releases1001 still syncing disks for a while (50m), being created... T164030 |
[production] |
| 23:49 |
<mutante> |
ganeti: removed instance mwreleases1001, created new instance releases1001 with same parameters (2 VCPUS,4G memory, 1 x 128G disk) (T164030) |
[production] |
| 23:41 |
<mutante> |
mwreleases1001 - scheduled downtime, shutdown, kill VM, re-install as releases1001 (T164030) |
[production] |
| 23:33 |
<catrope@tin> |
Synchronized php-1.30.0-wmf.5/includes/: Unbreak watchlist highlighting T167922 (duration: 01m 30s) |
[production] |
| 23:30 |
<catrope@tin> |
Synchronized wmf-config/InitialiseSettings.php: Send search traffic back to eqiad T149006 (duration: 00m 44s) |
[production] |
| 23:23 |
<catrope@tin> |
Synchronized wmf-config/: ORES config cleanups (duration: 00m 46s) |
[production] |