|
2020-10-20
ยง
|
| 20:39 |
<cdanis> |
doing some manual testing on mw2221, depooled and puppet disabled |
[production] |
| 20:33 |
<mforns@deploy1001> |
Finished deploy [analytics/refinery@e4d16f0]: Regular analytics weekly train [analytics/refinery@e4d16f08a96b6f65447fcdc6c9e8945724a89f54] (duration: 08m 10s) |
[production] |
| 20:31 |
<ryankemper> |
[Temporarily] disabled notifications for all wdqs hosts while we figure out how to unstick the updater process. Impact is that new updates will be delayed, but queries will still keep serving as normal, so fixing this is a priority but note that there's no availability outage |
[production] |
| 20:29 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 20:25 |
<mforns@deploy1001> |
Started deploy [analytics/refinery@e4d16f0]: Regular analytics weekly train [analytics/refinery@e4d16f08a96b6f65447fcdc6c9e8945724a89f54] |
[production] |
| 20:24 |
<mforns> |
Deploying refinery with scap for v0.0.137 |
[analytics] |
| 20:19 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 20:18 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 20:06 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 20:00 |
<mforns> |
Deployed refinery-source v0.0.137 |
[analytics] |
| 19:59 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 19:47 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 19:47 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 19:47 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 19:45 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: dc=codfw,cluster=parsoid,service=canary |
[production] |
| 19:24 |
<razzi@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 18:58 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 18:56 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:36 |
<bstorm> |
brought up mariadb and replication on clouddb1002 T263677 |
[clouddb-services] |
| 17:48 |
<effie> |
depooling mw2328 - T266052 |
[production] |
| 17:37 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:35 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:14 |
<bstorm> |
shutting down clouddb1003 T263677 |
[clouddb-services] |
| 17:13 |
<bstorm> |
stopping postgresql on clouddb1003 T263677 |
[clouddb-services] |
| 17:08 |
<bstorm> |
poweroff clouddb1002 T263677 |
[clouddb-services] |
| 17:08 |
<bstorm> |
stopping mariadb on clouddb1002 T263677 |
[clouddb-services] |
| 17:07 |
<bstorm> |
shut down replication on clouddb1002 (now with task) T263677 |
[clouddb-services] |
| 17:05 |
<bstorm> |
shut down replication on clouddb1002 |
[clouddb-services] |
| 16:11 |
<bstorm> |
restarted mariadb on quarry-db-01 so it pointed to the right data directory |
[quarry] |
| 16:00 |
<andrewbogott> |
rebooting quarry-web-01; lots of cruft in /tmp |
[quarry] |
| 15:56 |
<andrewbogott> |
restarting nginx on quarry-web-01 |
[quarry] |
| 15:54 |
<ebernhardson@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@629e8bc]: search satisfaction: remove unused y/m/d cli args (duration: 01m 31s) |
[production] |
| 15:52 |
<ebernhardson@deploy1001> |
Started deploy [wikimedia/discovery/analytics@629e8bc]: search satisfaction: remove unused y/m/d cli args |
[production] |
| 15:47 |
<arturo> |
changing DNS recursor ACLs (https://gerrit.wikimedia.org/r/c/operations/puppet/+/635314) this can be reverted any time if it causes problems (T261724) |
[admin] |
| 15:15 |
<aborrero@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:13 |
<aborrero@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:00 |
<ottomata> |
disabling sending EventLogging events to eventlogging-valid-mixed topic - T265651 |
[analytics] |
| 14:58 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.13/extensions/AbuseFilter/includes/Views/AbuseFilterViewList.php: fee2d3be13ae14d7ea51ff2db42090a1c27819bf: Prevent uncaught warnings/exception on Special:AbuseFilter (T265994) (duration: 01m 03s) |
[production] |
| 14:56 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.14/extensions/AbuseFilter/includes/Views/AbuseFilterViewList.php: 00ef00f59fd2a7a1366161ccc66c260be20e3e50: Prevent uncaught warnings/exception on Special:AbuseFilter (T265994) (duration: 01m 01s) |
[production] |
| 14:49 |
<arturo> |
[codfw1dev] reimaging labtestvirt2003 (cloudgw) to test puppet code (T261724) |
[admin] |
| 14:48 |
<wm-bot> |
<root> Deleted 74G ~/logs/commands.log log file |
[tools.smallem] |
| 14:48 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.14/extensions/FileImporter/: 5eee9b773338e5181867cabec9faefbdeacf67ca: Set originalRequest (incl. X-Forwarded-For) for remote edits (T265810) (duration: 01m 06s) |
[production] |
| 14:26 |
<hashar> |
Switched CI jobs to rust 1.47 https://gerrit.wikimedia.org/r/c/integration/config/+/633693 |
[releng] |
| 14:16 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.13/extensions/FileImporter/: 5f8d3de14c116b618f5226419082d5c9a07766fb: Set originalRequest (incl. X-Forwarded-For) for remote edits (T265810) (duration: 01m 09s) |
[production] |
| 14:15 |
<Urbanecm> |
[urbanecm@deploy1001 /srv/mediawiki-staging (master u=)]$ sudo /usr/local/sbin/fix-staging-perms |
[production] |
| 13:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2125 (re)pooling @ 100%: Slowly repool db2125 after checking tables ', diff saved to https://phabricator.wikimedia.org/P13033 and previous config saved to /var/cache/conftool/dbconfig/20201020-135436-root.json |
[production] |
| 13:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2125 (re)pooling @ 80%: Slowly repool db2125 after checking tables ', diff saved to https://phabricator.wikimedia.org/P13032 and previous config saved to /var/cache/conftool/dbconfig/20201020-133933-root.json |
[production] |
| 13:34 |
<elukey> |
upgrade superset's presto TLS config after the above changes |
[analytics] |
| 13:33 |
<elukey> |
move presto to pupet host TLS certificates |
[analytics] |
| 13:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2125 (re)pooling @ 60%: Slowly repool db2125 after checking tables ', diff saved to https://phabricator.wikimedia.org/P13031 and previous config saved to /var/cache/conftool/dbconfig/20201020-132430-root.json |
[production] |