|
2020-10-20
§
|
| 21:18 |
<cdanis> |
✔️ cdanis@mw2252.codfw.wmnet ~ 🕠🍺 sudo depool |
[production] |
| 20:57 |
<mforns@deploy1001> |
Finished deploy [analytics/refinery@e4d16f0] (thin): Regular analytics weekly train THIN [analytics/refinery@e4d16f08a96b6f65447fcdc6c9e8945724a89f54] (duration: 00m 08s) |
[production] |
| 20:56 |
<mforns@deploy1001> |
Started deploy [analytics/refinery@e4d16f0] (thin): Regular analytics weekly train THIN [analytics/refinery@e4d16f08a96b6f65447fcdc6c9e8945724a89f54] |
[production] |
| 20:39 |
<cdanis> |
doing some manual testing on mw2221, depooled and puppet disabled |
[production] |
| 20:33 |
<mforns@deploy1001> |
Finished deploy [analytics/refinery@e4d16f0]: Regular analytics weekly train [analytics/refinery@e4d16f08a96b6f65447fcdc6c9e8945724a89f54] (duration: 08m 10s) |
[production] |
| 20:31 |
<ryankemper> |
[Temporarily] disabled notifications for all wdqs hosts while we figure out how to unstick the updater process. Impact is that new updates will be delayed, but queries will still keep serving as normal, so fixing this is a priority but note that there's no availability outage |
[production] |
| 20:29 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 20:25 |
<mforns@deploy1001> |
Started deploy [analytics/refinery@e4d16f0]: Regular analytics weekly train [analytics/refinery@e4d16f08a96b6f65447fcdc6c9e8945724a89f54] |
[production] |
| 20:19 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 20:18 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 20:06 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 19:59 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 19:47 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 19:47 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 19:47 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 19:45 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: dc=codfw,cluster=parsoid,service=canary |
[production] |
| 19:24 |
<razzi@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 18:58 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 18:56 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:48 |
<effie> |
depooling mw2328 - T266052 |
[production] |
| 17:37 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:35 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:54 |
<ebernhardson@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@629e8bc]: search satisfaction: remove unused y/m/d cli args (duration: 01m 31s) |
[production] |
| 15:52 |
<ebernhardson@deploy1001> |
Started deploy [wikimedia/discovery/analytics@629e8bc]: search satisfaction: remove unused y/m/d cli args |
[production] |
| 15:15 |
<aborrero@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:13 |
<aborrero@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:58 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.13/extensions/AbuseFilter/includes/Views/AbuseFilterViewList.php: fee2d3be13ae14d7ea51ff2db42090a1c27819bf: Prevent uncaught warnings/exception on Special:AbuseFilter (T265994) (duration: 01m 03s) |
[production] |
| 14:56 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.14/extensions/AbuseFilter/includes/Views/AbuseFilterViewList.php: 00ef00f59fd2a7a1366161ccc66c260be20e3e50: Prevent uncaught warnings/exception on Special:AbuseFilter (T265994) (duration: 01m 01s) |
[production] |
| 14:48 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.14/extensions/FileImporter/: 5eee9b773338e5181867cabec9faefbdeacf67ca: Set originalRequest (incl. X-Forwarded-For) for remote edits (T265810) (duration: 01m 06s) |
[production] |
| 14:16 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.13/extensions/FileImporter/: 5f8d3de14c116b618f5226419082d5c9a07766fb: Set originalRequest (incl. X-Forwarded-For) for remote edits (T265810) (duration: 01m 09s) |
[production] |
| 14:15 |
<Urbanecm> |
[urbanecm@deploy1001 /srv/mediawiki-staging (master u=)]$ sudo /usr/local/sbin/fix-staging-perms |
[production] |
| 13:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2125 (re)pooling @ 100%: Slowly repool db2125 after checking tables ', diff saved to https://phabricator.wikimedia.org/P13033 and previous config saved to /var/cache/conftool/dbconfig/20201020-135436-root.json |
[production] |
| 13:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2125 (re)pooling @ 80%: Slowly repool db2125 after checking tables ', diff saved to https://phabricator.wikimedia.org/P13032 and previous config saved to /var/cache/conftool/dbconfig/20201020-133933-root.json |
[production] |
| 13:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2125 (re)pooling @ 60%: Slowly repool db2125 after checking tables ', diff saved to https://phabricator.wikimedia.org/P13031 and previous config saved to /var/cache/conftool/dbconfig/20201020-132430-root.json |
[production] |
| 13:19 |
<XioNoX> |
install routinator 3000 0.8.0 on rpki2001 - T266001 |
[production] |
| 13:16 |
<liw@deploy1001> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.36.0-wmf.14 |
[production] |
| 13:11 |
<liw@deploy1001> |
Finished scap: testwikis wikis to 1.36.0-wmf.14 (duration: 58m 03s) |
[production] |
| 13:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2125 (re)pooling @ 40%: Slowly repool db2125 after checking tables ', diff saved to https://phabricator.wikimedia.org/P13030 and previous config saved to /var/cache/conftool/dbconfig/20201020-130926-root.json |
[production] |
| 12:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2125 (re)pooling @ 20%: Slowly repool db2125 after checking tables ', diff saved to https://phabricator.wikimedia.org/P13029 and previous config saved to /var/cache/conftool/dbconfig/20201020-125423-root.json |
[production] |
| 12:25 |
<jayme@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 12:25 |
<jayme@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 12:24 |
<jayme@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 12:24 |
<jayme@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 12:16 |
<jayme@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 12:16 |
<jayme@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 12:15 |
<jayme@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 12:13 |
<liw@deploy1001> |
Started scap: testwikis wikis to 1.36.0-wmf.14 |
[production] |
| 11:37 |
<liw> |
1.36.0-wmf.14 was branched at 1b7b5f716015f9303d37158820dadf759e8db707 for T263180 |
[production] |
| 11:35 |
<Lucas_WMDE> |
EU backport/config window done |
[production] |
| 11:35 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized php-1.36.0-wmf.13/extensions/WikimediaEvents/: Backport: [[gerrit:635030|SearchSatisfaction: Set isAnon field (T259250)]] (duration: 00m 57s) |
[production] |