|
2020-07-14
§
|
| 08:30 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:13 |
<jforrester@deploy1001> |
Started scap: Re-re-start full scap to push out wmf.41 and switch testwikis to it T256669 |
[production] |
| 08:05 |
<akosiaris> |
restart pybal on lvs2009 |
[production] |
| 08:03 |
<_joe_> |
restart pybal on lvs1016 |
[production] |
| 08:02 |
<akosiaris> |
restart pybal on lvs2007 |
[production] |
| 08:01 |
<akosiaris@cumin1001> |
conftool action : set/pooled=inactive; selector: name=restbase2009.codfw.wmnet |
[production] |
| 08:00 |
<_joe_> |
restart pybal on lvs1015 |
[production] |
| 08:00 |
<akosiaris> |
restart pybal on lvs2010 after merging https://gerrit.wikimedia.org/r/612487 |
[production] |
| 07:52 |
<jforrester@deploy1001> |
sync aborted: Re-start full scap to push out wmf.41 and switch testwikis to it T256669 (duration: 02m 14s) |
[production] |
| 07:50 |
<jforrester@deploy1001> |
Started scap: Re-start full scap to push out wmf.41 and switch testwikis to it T256669 |
[production] |
| 07:48 |
<oblivian@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: revert forcehttps in an attempt to fix T257887 (duration: 01m 06s) |
[production] |
| 07:32 |
<oblivian@deploy1001> |
sync-file aborted: revert forcehttps in an attempt to fix T257887 (duration: 00m 20s) |
[production] |
| 07:31 |
<oblivian@deploy1001> |
Scap failed!: 7/9 canaries failed their endpoint checks(http://en.wikipedia.org) |
[production] |
| 07:27 |
<moritzm> |
installing libtasn1-6 security updates |
[production] |
| 07:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1075', diff saved to https://phabricator.wikimedia.org/P11894 and previous config saved to /var/cache/conftool/dbconfig/20200714-071233-marostegui.json |
[production] |
| 07:04 |
<marostegui> |
Drop gerrit, gerritro, gerrittest users from m2 databases - T255715 |
[production] |
| 06:58 |
<marostegui> |
Stop mysql on db1131 for HW maintenance |
[production] |
| 06:56 |
<oblivian@deploy2001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 06:54 |
<jforrester@deploy1001> |
scap failed: RuntimeError Scap failed!: 9/9 canaries failed their endpoint checks(http://en.wikipedia.org) (duration: 24m 59s) |
[production] |
| 06:54 |
<jforrester@deploy1001> |
Scap failed!: 9/9 canaries failed their endpoint checks(http://en.wikipedia.org) |
[production] |
| 06:53 |
<oblivian@deploy2001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 06:53 |
<marostegui> |
Deploy MCR schema change on s5 primary master T238966 |
[production] |
| 06:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1078', diff saved to https://phabricator.wikimedia.org/P11893 and previous config saved to /var/cache/conftool/dbconfig/20200714-065229-marostegui.json |
[production] |
| 06:29 |
<jforrester@deploy1001> |
Started scap: testwikis wikis to 1.35.0-wmf.41 |
[production] |
| 05:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Decrease a bit db1088 load', diff saved to https://phabricator.wikimedia.org/P11891 and previous config saved to /var/cache/conftool/dbconfig/20200714-051551-marostegui.json |
[production] |
| 05:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1131 for HW maintenance', diff saved to https://phabricator.wikimedia.org/P11890 and previous config saved to /var/cache/conftool/dbconfig/20200714-050931-marostegui.json |
[production] |
| 05:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1093 from api', diff saved to https://phabricator.wikimedia.org/P11889 and previous config saved to /var/cache/conftool/dbconfig/20200714-050912-marostegui.json |
[production] |
| 05:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db1093 to s6 master and remove read-only from s6 T257253', diff saved to https://phabricator.wikimedia.org/P11888 and previous config saved to /var/cache/conftool/dbconfig/20200714-050157-marostegui.json |
[production] |
| 05:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set s6 as read-only for maintenance T257253', diff saved to https://phabricator.wikimedia.org/P11887 and previous config saved to /var/cache/conftool/dbconfig/20200714-050039-marostegui.json |
[production] |
| 05:00 |
<marostegui> |
Starting s6 failover from db1131 to db1093 - T257253 |
[production] |
| 04:59 |
<James_F> |
1.35.0-wmf.41 branched at 7d04152db4f8ea9a459511bed8117101d9bb4602 |
[production] |
| 04:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1078', diff saved to https://phabricator.wikimedia.org/P11886 and previous config saved to /var/cache/conftool/dbconfig/20200714-043907-marostegui.json |
[production] |
| 04:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1093 in preparation for failover', diff saved to https://phabricator.wikimedia.org/P11885 and previous config saved to /var/cache/conftool/dbconfig/20200714-041548-marostegui.json |
[production] |
| 04:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1130', diff saved to https://phabricator.wikimedia.org/P11884 and previous config saved to /var/cache/conftool/dbconfig/20200714-041440-marostegui.json |
[production] |
| 01:23 |
<ryankemper> |
Started long-running Elasticsearch reindex of `eqiad`, `codfw`, and `cloudelastic`. tmux session `reindex` under `ryankemper` on `mwmaint1002` |
[production] |
| 01:20 |
<cdanis> |
❌cdanis@lvs1015.eqiad.wmnet ~ 🕤🍺 sudo systemctl restart pybal.service |
[production] |
| 01:15 |
<cdanis> |
✔️ cdanis@lvs1016.eqiad.wmnet ~ 🕘🍺 sudo systemctl restart pybal.service |
[production] |
| 01:14 |
<cdanis> |
✔️ cdanis@lvs2009.codfw.wmnet ~ 🕘🍺 sudo systemctl restart pybal.service |
[production] |
| 01:01 |
<cdanis> |
✔️ cdanis@lvs2010.codfw.wmnet ~ 🕘🍺 sudo systemctl restart pybal.service |
[production] |
|
2020-07-13
§
|
| 23:06 |
<mutante> |
releases* delete /usr/local/sbin/sync-* scripts created by rsync::quickdatacopy and let puppet recreate the ones still needed |
[production] |
| 22:27 |
<krinkle@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: I80ca62643f5c (duration: 00m 58s) |
[production] |
| 20:12 |
<ebernhardson@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@1edde21]: airflow: ship_to_es: Implement multi-index understanding (duration: 00m 29s) |
[production] |
| 20:12 |
<ebernhardson@deploy1001> |
Started deploy [wikimedia/discovery/analytics@1edde21]: airflow: ship_to_es: Implement multi-index understanding |
[production] |
| 20:03 |
<mutante> |
rsynced reprepro data from releases1001 to releases1002, releases2002 |
[production] |
| 19:50 |
<eileen> |
disable target smart job process-control config revision is b00e7680ca |
[production] |
| 19:48 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@de0a1f1] (thin): Regular analytics weekly train THIN [analytics/refinery@de0a1f1] (duration: 00m 07s) |
[production] |
| 19:47 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@de0a1f1] (thin): Regular analytics weekly train THIN [analytics/refinery@de0a1f1] |
[production] |
| 19:47 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@de0a1f1]: Regular analytics weekly train [analytics/refinery@de0a1f1] (duration: 06m 41s) |
[production] |
| 19:41 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@de0a1f1]: Regular analytics weekly train [analytics/refinery@de0a1f1] |
[production] |
| 19:39 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |