|
2019-09-11
§
|
| 08:07 |
<jmm@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1122', diff saved to https://phabricator.wikimedia.org/P9088 and previous config saved to /var/cache/conftool/dbconfig/20190911-080450-marostegui.json |
[production] |
| 07:52 |
<moritzm> |
reimaging restbase-dev1005 to Stretch T224554 |
[production] |
| 07:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'More traffic to db1122', diff saved to https://phabricator.wikimedia.org/P9087 and previous config saved to /var/cache/conftool/dbconfig/20190911-075139-marostegui.json |
[production] |
| 07:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'More traffic to db1122', diff saved to https://phabricator.wikimedia.org/P9086 and previous config saved to /var/cache/conftool/dbconfig/20190911-073335-marostegui.json |
[production] |
| 07:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1122', diff saved to https://phabricator.wikimedia.org/P9085 and previous config saved to /var/cache/conftool/dbconfig/20190911-072344-marostegui.json |
[production] |
| 07:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1122', diff saved to https://phabricator.wikimedia.org/P9084 and previous config saved to /var/cache/conftool/dbconfig/20190911-071450-marostegui.json |
[production] |
| 07:07 |
<marostegui> |
Stop MySQL on db1122 to reboot for a kernel upgrade T230785 |
[production] |
| 07:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1122 to reboot for kernel upgrade T230785', diff saved to https://phabricator.wikimedia.org/P9083 and previous config saved to /var/cache/conftool/dbconfig/20190911-070635-marostegui.json |
[production] |
| 07:00 |
<hashar> |
Restarting Gerrit - T224448 |
[production] |
| 06:58 |
<hashar> |
Restarting Gerrit |
[production] |
| 06:45 |
<marostegui> |
Drop unused database puppet on m1 - T231539 |
[production] |
| 06:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Re-organize s1 codfw weights and roles - T230106', diff saved to https://phabricator.wikimedia.org/P9082 and previous config saved to /var/cache/conftool/dbconfig/20190911-061924-marostegui.json |
[production] |
| 06:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Re-organize s1 codfw weights and roles - T230106', diff saved to https://phabricator.wikimedia.org/P9081 and previous config saved to /var/cache/conftool/dbconfig/20190911-061659-marostegui.json |
[production] |
| 05:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2048, will be decommissioned T230106', diff saved to https://phabricator.wikimedia.org/P9080 and previous config saved to /var/cache/conftool/dbconfig/20190911-054855-marostegui.json |
[production] |
| 05:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db2112 to s1 codfw master T230106', diff saved to https://phabricator.wikimedia.org/P9079 and previous config saved to /var/cache/conftool/dbconfig/20190911-054753-marostegui.json |
[production] |
| 05:29 |
<marostegui> |
Switchover s1 codfw master db2048 -> db2112 T230106 |
[production] |
| 03:31 |
<eileen> |
civicrm revision changed from b343642c76 to 53aeba6318, config revision is 3e22a80bc8 |
[production] |
|
2019-09-10
§
|
| 20:46 |
<ejegg> |
updated payments-wiki from 15baf7f58b to 5432f9c3a4 |
[production] |
| 20:24 |
<XioNoX> |
add MSS clamp on install1002 - T2324563 |
[production] |
| 20:20 |
<XioNoX> |
add MSS clamp on archiva1001 - T232456 |
[production] |
| 18:42 |
<herron> |
rolling out "Aggregate IPsec Tunnel Status” icinga check, please disregard for the time being if it alerts |
[production] |
| 18:15 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T229863 Remove EventBusRCFeedEngine eventServiceName (duration: 01m 05s) |
[production] |
| 18:15 |
<XioNoX> |
rollback test add static route on bast3002 to force advmss |
[production] |
| 18:10 |
<XioNoX> |
test add static route on bast3002 to force advmss |
[production] |
| 17:58 |
<jforrester@deploy1001> |
Synchronized wmf-config/logging.php: T232042 Direct Parsoid/PHP rt-testing log events to a different target (duration: 01m 02s) |
[production] |
| 17:56 |
<jforrester@deploy1001> |
Synchronized wmf-config/ProductionServices.php: T232122 Stop setting production value for eventlogging-service (duration: 01m 00s) |
[production] |
| 17:55 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T232122 Remove use of eventlogging-service (duration: 01m 03s) |
[production] |
| 17:33 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Re-sync for safety after scap errored with a broken pipe (duration: 01m 03s) |
[production] |
| 17:31 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Variant configuration: Write to static (JSON) as well as serialised cache for testwiki T223602 (duration: 01m 02s) |
[production] |
| 17:29 |
<jforrester@deploy1001> |
Synchronized multiversion/MWConfigCacheGenerator.php: Variant configuration: Be able to write to static (JSON) as well as serialised cache (duration: 01m 03s) |
[production] |
| 16:35 |
<elukey> |
reboot analytics-tool1001 via ganeti gnt - not reachable via ssh |
[production] |
| 16:24 |
<urandom> |
disabling reserved space on restbase-dev1005:/dev/mapper/restbase--dev1005--vg-srv -- T224554 |
[production] |
| 16:10 |
<marostegui> |
Failover m1 from db1063 to db1135 - T231403 |
[production] |
| 15:58 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Revert "Set items term store on write both for all of Wikidata" (duration: 01m 02s) |
[production] |
| 15:58 |
<thcipriani> |
restarting gerrit (again) https://grafana.wikimedia.org/d/Bw2mQ3iWz/gerrit-javamelody?orgId=1&from=1568109359163&to=1568130959163&var-Application=&var-Window=30m due to T224448 |
[production] |
| 15:39 |
<hashar@deploy1001> |
rebuilt and synchronized wikiversions files: group0 to 1.34.0-wmf.22 |
[production] |
| 15:37 |
<marostegui> |
Start pre-switchover for m1 steps T231403 |
[production] |
| 15:35 |
<hashar@deploy1001> |
Synchronized php-1.34.0-wmf.22/includes/libs/http/MultiHttpClient.php: Revert "Improve MultiHttpClient connection concurrency and reuse" - T232487 (duration: 00m 55s) |
[production] |
| 15:33 |
<reedy@deploy1001> |
Synchronized php-1.34.0-wmf.22/includes/libs/http/MultiHttpClient.php: T232487 (duration: 00m 55s) |
[production] |
| 15:13 |
<hashar@deploy1001> |
rebuilt and synchronized wikiversions files: Revert group0 to 1.34.0-wmf.22 # T220747 |
[production] |
| 14:48 |
<hashar@deploy1001> |
scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) |
[production] |
| 14:45 |
<akosiaris> |
repool cp1075 ats-be, releases cert updated |
[production] |
| 14:44 |
<akosiaris@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=cp1075.eqiad.wmnet,dc=eqiad,cluster=cache_text,service=ats-be |
[production] |
| 14:44 |
<XioNoX> |
depool ulsfo for DC UPS power maintenance (see maint-announce) |
[production] |
| 14:36 |
<@> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-main' for release 'main' . |
[production] |
| 14:32 |
<hashar@deploy1001> |
Finished scap: testwiki to php-1.34.0-wmf.22 and rebuild l10n cache # T220747 (duration: 34m 03s) |
[production] |
| 14:31 |
<@> |
helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-main' for release 'main' . |
[production] |
| 14:29 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-main' for release 'main' . |
[production] |
| 14:26 |
<@> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-analytics' for release 'analytics' . |
[production] |