|
2020-07-14
§
|
| 07:32 |
<oblivian@deploy1001> |
sync-file aborted: revert forcehttps in an attempt to fix T257887 (duration: 00m 20s) |
[production] |
| 07:31 |
<oblivian@deploy1001> |
Scap failed!: 7/9 canaries failed their endpoint checks(http://en.wikipedia.org) |
[production] |
| 07:27 |
<joal> |
Restart forgotten unique-devices per-project-family jobs after yesterday deploy |
[analytics] |
| 07:27 |
<moritzm> |
installing libtasn1-6 security updates |
[production] |
| 07:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1075', diff saved to https://phabricator.wikimedia.org/P11894 and previous config saved to /var/cache/conftool/dbconfig/20200714-071233-marostegui.json |
[production] |
| 07:04 |
<marostegui> |
Drop gerrit, gerritro, gerrittest users from m2 databases - T255715 |
[production] |
| 06:58 |
<marostegui> |
Stop mysql on db1131 for HW maintenance |
[production] |
| 06:56 |
<oblivian@deploy2001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 06:54 |
<jforrester@deploy1001> |
scap failed: RuntimeError Scap failed!: 9/9 canaries failed their endpoint checks(http://en.wikipedia.org) (duration: 24m 59s) |
[production] |
| 06:54 |
<jforrester@deploy1001> |
Scap failed!: 9/9 canaries failed their endpoint checks(http://en.wikipedia.org) |
[production] |
| 06:53 |
<oblivian@deploy2001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 06:53 |
<marostegui> |
Deploy MCR schema change on s5 primary master T238966 |
[production] |
| 06:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1078', diff saved to https://phabricator.wikimedia.org/P11893 and previous config saved to /var/cache/conftool/dbconfig/20200714-065229-marostegui.json |
[production] |
| 06:29 |
<jforrester@deploy1001> |
Started scap: testwikis wikis to 1.35.0-wmf.41 |
[production] |
| 05:44 |
<James_F> |
Created WMF-deployed REL1_35 branches with `./branch.py --branchpoint 7d04152db4f8ea9a459511bed8117101d9bb4602 --bundle base --core-version 1.35.0-beta --task T256376 REL1_35` |
[releng] |
| 05:17 |
<James_F> |
Zuul: parameter_functions: VisualEditor and TemplateData are now tarballed (no-op) |
[releng] |
| 05:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Decrease a bit db1088 load', diff saved to https://phabricator.wikimedia.org/P11891 and previous config saved to /var/cache/conftool/dbconfig/20200714-051551-marostegui.json |
[production] |
| 05:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1131 for HW maintenance', diff saved to https://phabricator.wikimedia.org/P11890 and previous config saved to /var/cache/conftool/dbconfig/20200714-050931-marostegui.json |
[production] |
| 05:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1093 from api', diff saved to https://phabricator.wikimedia.org/P11889 and previous config saved to /var/cache/conftool/dbconfig/20200714-050912-marostegui.json |
[production] |
| 05:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db1093 to s6 master and remove read-only from s6 T257253', diff saved to https://phabricator.wikimedia.org/P11888 and previous config saved to /var/cache/conftool/dbconfig/20200714-050157-marostegui.json |
[production] |
| 05:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set s6 as read-only for maintenance T257253', diff saved to https://phabricator.wikimedia.org/P11887 and previous config saved to /var/cache/conftool/dbconfig/20200714-050039-marostegui.json |
[production] |
| 05:00 |
<marostegui> |
Starting s6 failover from db1131 to db1093 - T257253 |
[production] |
| 04:59 |
<James_F> |
1.35.0-wmf.41 branched at 7d04152db4f8ea9a459511bed8117101d9bb4602 |
[production] |
| 04:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1078', diff saved to https://phabricator.wikimedia.org/P11886 and previous config saved to /var/cache/conftool/dbconfig/20200714-043907-marostegui.json |
[production] |
| 04:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1093 in preparation for failover', diff saved to https://phabricator.wikimedia.org/P11885 and previous config saved to /var/cache/conftool/dbconfig/20200714-041548-marostegui.json |
[production] |
| 04:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1130', diff saved to https://phabricator.wikimedia.org/P11884 and previous config saved to /var/cache/conftool/dbconfig/20200714-041440-marostegui.json |
[production] |
| 04:01 |
<andrewbogott> |
added a wildcard *.wmflabs.org domain pointing at the domain proxy in project-proxy |
[admin] |
| 04:00 |
<andrewbogott> |
shortened the ttl on .wmflabs.org. to 300 |
[admin] |
| 01:23 |
<ryankemper> |
Started long-running Elasticsearch reindex of `eqiad`, `codfw`, and `cloudelastic`. tmux session `reindex` under `ryankemper` on `mwmaint1002` |
[production] |
| 01:20 |
<cdanis> |
❌cdanis@lvs1015.eqiad.wmnet ~ 🕤🍺 sudo systemctl restart pybal.service |
[production] |
| 01:15 |
<cdanis> |
✔️ cdanis@lvs1016.eqiad.wmnet ~ 🕘🍺 sudo systemctl restart pybal.service |
[production] |
| 01:14 |
<cdanis> |
✔️ cdanis@lvs2009.codfw.wmnet ~ 🕘🍺 sudo systemctl restart pybal.service |
[production] |
| 01:01 |
<cdanis> |
✔️ cdanis@lvs2010.codfw.wmnet ~ 🕘🍺 sudo systemctl restart pybal.service |
[production] |
|
2020-07-13
§
|
| 23:06 |
<mutante> |
releases* delete /usr/local/sbin/sync-* scripts created by rsync::quickdatacopy and let puppet recreate the ones still needed |
[production] |
| 22:27 |
<krinkle@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: I80ca62643f5c (duration: 00m 58s) |
[production] |
| 22:12 |
<bd808> |
Add codesearch.wmflabs.org (T256206) |
[redirects] |
| 21:04 |
<Amir1> |
create codesearch.wmcloud.org dns proxy (T256206) |
[codesearch] |
| 20:53 |
<Reedy> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/612364 |
[releng] |
| 20:17 |
<milimetric> |
deployed weekly train with two oozie job bugfixes and rename to pageview_actor table |
[analytics] |
| 20:12 |
<ebernhardson@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@1edde21]: airflow: ship_to_es: Implement multi-index understanding (duration: 00m 29s) |
[production] |
| 20:12 |
<ebernhardson@deploy1001> |
Started deploy [wikimedia/discovery/analytics@1edde21]: airflow: ship_to_es: Implement multi-index understanding |
[production] |
| 20:03 |
<mutante> |
rsynced reprepro data from releases1001 to releases1002, releases2002 |
[production] |
| 19:50 |
<eileen> |
disable target smart job process-control config revision is b00e7680ca |
[production] |
| 19:48 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@de0a1f1] (thin): Regular analytics weekly train THIN [analytics/refinery@de0a1f1] (duration: 00m 07s) |
[production] |
| 19:47 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@de0a1f1] (thin): Regular analytics weekly train THIN [analytics/refinery@de0a1f1] |
[production] |
| 19:47 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@de0a1f1]: Regular analytics weekly train [analytics/refinery@de0a1f1] (duration: 06m 41s) |
[production] |
| 19:42 |
<joal> |
Deploy refinery with scap |
[analytics] |
| 19:41 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@de0a1f1]: Regular analytics weekly train [analytics/refinery@de0a1f1] |
[production] |
| 19:39 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 19:33 |
<krinkle@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: I1a12124f1811e9a (duration: 00m 57s) |
[production] |