|
2020-05-28
§
|
| 08:46 |
<XioNoX> |
deactivate peering/transit on cr2-eqord - T243080 |
[production] |
| 08:45 |
<XioNoX> |
de-pref all OSPF links to cr2-eqord - T243080 |
[production] |
| 08:13 |
<marostegui> |
Pool db1141 into labsdb analytics role - T249188 |
[production] |
| 07:33 |
<gilles@deploy1001> |
Synchronized static/images: T252108 Deploying optimised static PNGs (duration: 01m 39s) |
[production] |
| 07:31 |
<gilles@deploy1001> |
Synchronized static/apple-touch: T252108 Deploying optimised static PNGs (duration: 01m 12s) |
[production] |
| 06:40 |
<elukey> |
slowly restarting all RU units on an-launcher1001 |
[analytics] |
| 06:32 |
<elukey> |
delete old RU pid files with timestamp May 27 19:00 (scap deployment failed to an-launcher due to disk issues) except ./jobs/reportupdater-queries/pingback/.reportupdater.pid that was working fine |
[analytics] |
| 06:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1081 from API and set its weight to 0 on main traffic - preparation for tomorrow's failover T253808', diff saved to https://phabricator.wikimedia.org/P11329 and previous config saved to /var/cache/conftool/dbconfig/20200528-063037-marostegui.json |
[production] |
| 04:44 |
<marostegui> |
Run check_private data on db1141 - T249188 |
[production] |
| 04:22 |
<marostegui> |
Stop MySQL on db1141 - T249188 |
[production] |
| 00:33 |
<andrewbogott> |
shutting down cloudservices2002-dev to see if we can live without it. This is in anticipation or rebuilding it entirely for T253780 |
[admin] |
|
2020-05-27
§
|
| 23:29 |
<andrewbogott> |
disabling the backup job on cloudbackup2001 (just like last week) so the backup doesn't start while Brooke is rebuilding labstore1004 tomorrow. |
[admin] |
| 23:20 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Add autoreviewrestore right to rollbacker group on hiwiki (T252986) (duration: 01m 05s) |
[production] |
| 23:16 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Add thwiki Draft namespace to wmgExemptFromUserRobotsControlExtra and enable VE there (T252959) (duration: 01m 06s) |
[production] |
| 22:58 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.postgresql.postgres-init (exit_code=0) |
[production] |
| 22:02 |
<crusnov@deploy1001> |
Finished deploy [netbox/deploy@5251cf1]: Netbox Upgrade to 2.8.4 (part4) (duration: 00m 10s) |
[production] |
| 22:02 |
<crusnov@deploy1001> |
Started deploy [netbox/deploy@5251cf1]: Netbox Upgrade to 2.8.4 (part4) |
[production] |
| 22:01 |
<crusnov@deploy1001> |
Finished deploy [netbox/deploy@5251cf1]: Netbox Upgrade to 2.8.4 (part3) (duration: 01m 29s) |
[production] |
| 22:00 |
<crusnov@deploy1001> |
Started deploy [netbox/deploy@5251cf1]: Netbox Upgrade to 2.8.4 (part3) |
[production] |
| 22:00 |
<crusnov@deploy1001> |
deploy aborted: Netbox Upgrade to 2.8.4 (part2) (duration: 01m 31s) |
[production] |
| 21:58 |
<crusnov@deploy1001> |
Started deploy [netbox/deploy@5251cf1]: Netbox Upgrade to 2.8.4 (part2) |
[production] |
| 21:58 |
<crusnov@deploy1001> |
Finished deploy [netbox/deploy@5251cf1]: Netbox Upgrade to 2.8.1 (part1) (duration: 01m 01s) |
[production] |
| 21:57 |
<crusnov@deploy1001> |
Started deploy [netbox/deploy@5251cf1]: Netbox Upgrade to 2.8.1 (part1) |
[production] |
| 21:55 |
<James_F> |
Nicely restarting Jenkins for xunit plugin upgrade. |
[releng] |
| 20:43 |
<gehel@cumin1001> |
START - Cookbook sre.postgresql.postgres-init |
[production] |
| 20:28 |
<marostegui> |
Decrease innodb poolsize on s4 master and restart mysql |
[production] |
| 20:11 |
<mbsantos@deploy1001> |
Finished deploy [mobileapps/deploy@9dc827f]: Update mobileapps to b3b9214c (T253648) (duration: 03m 31s) |
[production] |
| 20:08 |
<mbsantos@deploy1001> |
Started deploy [mobileapps/deploy@9dc827f]: Update mobileapps to b3b9214c (T253648) |
[production] |
| 20:04 |
<twentyafterfour@deploy1001> |
Synchronized php: group1 wikis to 1.35.0-wmf.32 refs T253022 (duration: 01m 04s) |
[production] |
| 20:03 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.32 refs T253022 |
[production] |
| 20:00 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.postgresql.postgres-init (exit_code=0) |
[production] |
| 19:56 |
<twentyafterfour@deploy1001> |
scap failed: average error rate on 4/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/e474f13ffac6b8c3bf919c4aeafc8c9b for details) |
[production] |
| 19:53 |
<joal> |
Start pageview-complete dump oozie job after deploy |
[analytics] |
| 19:46 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.34/includes/parser/CoreParserFunctions.php: T253725 Partially revert 'Fix impedance mismatch with Parser::getRevisionRecordObject()' (duration: 01m 05s) |
[production] |
| 19:24 |
<joal> |
Deploy refinery onto hdfs |
[analytics] |
| 19:22 |
<joal> |
restart failed services on an-launcher1001 |
[analytics] |
| 19:12 |
<joal@deploy1001> |
Finished deploy [analytics/refinery@8a3dcb3]: Analytics regular weekly train (an-launcher1001 only) [8a3dcb3] (duration: 06m 07s) |
[production] |
| 19:09 |
<jforrester@deploy1001> |
Synchronized dblists/mobilemainpagelegacy.dblist: T32405 Stop special casing the main page on mobile for twelve wikis (duration: 01m 05s) |
[production] |
| 19:06 |
<joal> |
Deploy refinery using scap to an-launcher1001 only |
[analytics] |
| 19:06 |
<joal@deploy1001> |
Started deploy [analytics/refinery@8a3dcb3]: Analytics regular weekly train (an-launcher1001 only) [8a3dcb3] |
[production] |
| 19:03 |
<joal@deploy1001> |
Finished deploy [analytics/refinery@8a3dcb3] (thin): Analytics regular weekly train THIN [8a3dcb3] (duration: 00m 08s) |
[production] |
| 19:03 |
<joal@deploy1001> |
Started deploy [analytics/refinery@8a3dcb3] (thin): Analytics regular weekly train THIN [8a3dcb3] |
[production] |
| 19:03 |
<joal@deploy1001> |
Finished deploy [analytics/refinery@8a3dcb3]: Analytics regular weekly train [8a3dcb3] (duration: 21m 20s) |
[production] |
| 18:41 |
<joal> |
Deploying refinery with scap |
[analytics] |
| 18:41 |
<joal@deploy1001> |
Started deploy [analytics/refinery@8a3dcb3]: Analytics regular weekly train [8a3dcb3] |
[production] |
| 18:06 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Enable DiscussionTools as beta on mediawiki.org, part II T251208 (duration: 01m 05s) |
[production] |
| 17:56 |
<jayme> |
updated tiller to 2.16.7-wmf1 for all services in kubernetes cluster: eqiad |
[production] |
| 17:53 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Enable DiscussionTools as beta on mediawiki.org T251208 (duration: 01m 05s) |
[production] |
| 17:42 |
<gehel@cumin1001> |
START - Cookbook sre.postgresql.postgres-init |
[production] |
| 17:40 |
<gehel> |
repool maps2003 |
[production] |