|
2021-11-03
§
|
| 09:23 |
<XioNoX> |
re-enable eqiad Equinix IXP peerings - T290877 |
[production] |
| 08:55 |
<XioNoX> |
Disable eqiad Equinix IXP peerings - T290877 |
[production] |
| 07:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove logpager replicas from s6 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P17660 and previous config saved to /var/cache/conftool/dbconfig/20211103-075801-marostegui.json |
[production] |
| 07:58 |
<elukey@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 07:57 |
<elukey@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 07:50 |
<marostegui> |
Drop oauth2_access_tokens oauth_accepted_consumer oauth_registered_consumer from foundationwiki T294595 |
[production] |
| 06:43 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1163.eqiad.wmnet with OS buster |
[production] |
| 06:39 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 06:35 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 06:35 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 34888b034e54ec35ca3b6745336fc0881e50c9b0: Growth IP research survey: Fix coverage (T294568) (duration: 01m 04s) |
[production] |
| 06:13 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host db1163.eqiad.wmnet with OS buster |
[production] |
| 06:10 |
<marostegui> |
Stop replication on db1163 T290865 |
[production] |
| 06:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1163 until it's reimaged to buster T293964', diff saved to https://phabricator.wikimedia.org/P17659 and previous config saved to /var/cache/conftool/dbconfig/20211103-060644-root.json |
[production] |
| 06:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db1118 to s1 primary and set section read-write T293964', diff saved to https://phabricator.wikimedia.org/P17658 and previous config saved to /var/cache/conftool/dbconfig/20211103-060201-root.json |
[production] |
| 06:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set s1 eqiad as read-only for maintenance - T293964', diff saved to https://phabricator.wikimedia.org/P17657 and previous config saved to /var/cache/conftool/dbconfig/20211103-060114-root.json |
[production] |
| 06:00 |
<marostegui> |
Starting s1 eqiad failover from db1163 to db1118 - T293964 |
[production] |
| 05:01 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s1 T293964 |
[production] |
| 05:01 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 32 hosts with reason: Primary switchover s1 T293964 |
[production] |
| 02:22 |
<milimetric@deploy1002> |
Finished deploy [analytics/refinery@cf6095c] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@cf6095c] (duration: 05m 36s) |
[production] |
| 02:16 |
<milimetric@deploy1002> |
Started deploy [analytics/refinery@cf6095c] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@cf6095c] |
[production] |
| 02:16 |
<milimetric@deploy1002> |
Finished deploy [analytics/refinery@cf6095c] (thin): Regular analytics weekly train THIN [analytics/refinery@cf6095c] (duration: 00m 07s) |
[production] |
| 02:16 |
<milimetric@deploy1002> |
Started deploy [analytics/refinery@cf6095c] (thin): Regular analytics weekly train THIN [analytics/refinery@cf6095c] |
[production] |
| 02:15 |
<milimetric@deploy1002> |
Finished deploy [analytics/refinery@cf6095c]: Regular analytics weekly train [analytics/refinery@cf6095c] (duration: 22m 30s) |
[production] |
| 01:53 |
<milimetric@deploy1002> |
Started deploy [analytics/refinery@cf6095c]: Regular analytics weekly train [analytics/refinery@cf6095c] |
[production] |
|
2021-11-02
§
|
| 23:47 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 23:46 |
<tgr> |
UTC late deploys done |
[production] |
| 23:45 |
<tgr@deploy1002> |
Synchronized wmf-config: Config: Use page id for GrowthExperiments image recommendations, except for testwiki ([[gerrit:736314|736314]] [[gerrit:736317|736317]] (T290949 T292154) (duration: 01m 03s) |
[production] |
| 23:44 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 23:34 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 23:34 |
<tgr@deploy1002> |
Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:735094|Use url-downloader proxy for GrowthExperiments (T290949)]] (duration: 01m 14s) |
[production] |
| 23:30 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 22:14 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-db1002.eqiad.wmnet with OS buster |
[production] |
| 21:50 |
<robh@cumin1001> |
START - Cookbook sre.hosts.reimage for host an-db1002.eqiad.wmnet with OS buster |
[production] |
| 21:32 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-db1002.eqiad.wmnet with OS buster |
[production] |
| 21:03 |
<robh@cumin1001> |
START - Cookbook sre.hosts.reimage for host an-db1002.eqiad.wmnet with OS buster |
[production] |
| 20:52 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-db1001.eqiad.wmnet with OS buster |
[production] |
| 20:28 |
<robh@cumin1001> |
START - Cookbook sre.hosts.reimage for host an-db1001.eqiad.wmnet with OS buster |
[production] |
| 20:01 |
<thcipriani> |
1.38.0-wmf.7 on testwikis, leaving it there for today for US holiday (T293948) |
[production] |
| 19:58 |
<thcipriani@deploy1002> |
Pruned MediaWiki: 1.38.0-wmf.5 (duration: 04m 08s) |
[production] |
| 19:53 |
<thcipriani@deploy1002> |
Finished scap: testwikis wikis to 1.38.0-wmf.7 refs T293948 (duration: 50m 13s) |
[production] |
| 19:50 |
<moritzm> |
imported ganeti 2.16.0-1~bpo9+1+wmf1to component/ganeti216 for stretch-wikimedia (with additional cherrypicked patches for compat with KVM 3.1) T284811 |
[production] |
| 19:47 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 19:39 |
<robh@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 19:35 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts an-db1002.eqiad.wmnet |
[production] |
| 19:08 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:08 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-db1001.eqiad.wmnet with OS buster |
[production] |
| 19:05 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:02 |
<thcipriani@deploy1002> |
Started scap: testwikis wikis to 1.38.0-wmf.7 refs T293948 |
[production] |
| 18:46 |
<thcipriani> |
starting to stage train for 1.38.0-wmf.7 (T293948) |
[production] |
| 18:33 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts an-db1002.eqiad.wmnet |
[production] |