|
2019-08-23
§
|
| 17:40 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 17:39 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 12:24 |
<_joe_> |
pooling mw1270 temporarily, debugging performance issues |
[production] |
| 12:15 |
<_joe_> |
depooling mw1270 temporarily, performance issues |
[production] |
| 11:13 |
<marostegui> |
Upgrade db1114 from 10.3.16 to 10.3.17 |
[production] |
| 10:06 |
<dcausse> |
elastic: reindexing wikis with old mappings in eqiad & codfw (T230990) |
[production] |
| 05:52 |
<moritzm> |
installing squid3 security updates |
[production] |
| 05:11 |
<marostegui> |
Stop MySQL on db2066 for decommission T230885 |
[production] |
| 05:08 |
<marostegui> |
Remove db2066 from tendril and zarcillo T230885 |
[production] |
|
2019-08-22
§
|
| 23:41 |
<urbanecm@deploy1001> |
Synchronized php-1.34.0-wmf.19/extensions/WikibaseLexeme: SWAT: e4a5457: Fix Lexemes RDF generation (T230974) (duration: 00m 49s) |
[production] |
| 23:32 |
<urbanecm@deploy1001> |
Synchronized wmf-config/abusefilter.php: SWAT: eb1c4ea: Rename globals and rights in AbuseFilter config (duration: 00m 47s) |
[production] |
| 23:24 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 66b719d: General cleanup of `groupOverrides` (T231041) (duration: 00m 47s) |
[production] |
| 23:15 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 872f4b0: Change language code for punjabiwikimedia, resyncing, got broken pipe at the end (T230680) (duration: 00m 47s) |
[production] |
| 23:13 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 872f4b0: Change language code for punjabiwikimedia (T230680) (duration: 00m 48s) |
[production] |
| 23:10 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: a5917e4: Clean up `wgRateLimits` to remove unneeded entries (T231040) (duration: 00m 48s) |
[production] |
| 22:07 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: Update MachineVision Beta config (duration: 00m 47s) |
[production] |
| 21:19 |
<eileen> |
tools revision changed from 5c080bac63 to c0f4e7a379 |
[production] |
| 20:35 |
<ejegg> |
updated payments-wiki from 85dce8f79f to 231b7b0850 |
[production] |
| 17:14 |
<elukey> |
remove analytics-tool1002 from ganeti - T231021 |
[production] |
| 17:12 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 17:12 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 14:37 |
<_joe_> |
restarting php-fpm on mw1348 to observe the effect on the slowdown, T231011 |
[production] |
| 13:47 |
<elukey> |
update puppet compiler's facts |
[production] |
| 13:42 |
<jijiki> |
Restart php-fpm on mw1348 and mw1347 |
[production] |
| 13:41 |
<zfilipin@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.34.0-wmf.19 |
[production] |
| 13:36 |
<joal@deploy1001> |
Finished deploy [analytics/refinery@a9b99e9]: Regular weekly analytics deployment train (1 day late) (duration: 18m 57s) |
[production] |
| 13:32 |
<jiji@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Reverting PHP7 traffic back to 20% - T219150 (duration: 00m 57s) |
[production] |
| 13:27 |
<tarrow@deploy1001> |
Synchronized php-1.34.0-wmf.19/extensions/Wikibase/client/: [[gerrit:531677|Revert "Use the backwards-compatible HTML ID for the wikidata item link" (T230958, T66315)]] (duration: 00m 58s) |
[production] |
| 13:18 |
<joal@deploy1001> |
Started deploy [analytics/refinery@a9b99e9]: Regular weekly analytics deployment train (1 day late) |
[production] |
| 12:56 |
<_joe_> |
restarting mw1270 with slowlog disabled |
[production] |
| 12:41 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:40 |
<elukey@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 12:39 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:39 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:38 |
<_joe_> |
disabled slowlog on mw1348, repooling after reload |
[production] |
| 12:37 |
<jijiki> |
Pooling mv1347 not mw1247 |
[production] |
| 12:35 |
<jijiki> |
Pooling mw1247 |
[production] |
| 12:16 |
<moritzm> |
upgrading mariadb (packaged Debian version) on matomo1001 |
[production] |
| 12:15 |
<elukey@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 12:13 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:07 |
<jijiki> |
Depooling mw1347 and mw1348 |
[production] |
| 10:55 |
<jiji@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Push PHP7 traffic to 33.3% - T219150 (duration: 01m 01s) |
[production] |
| 09:09 |
<ema> |
rolling ats-backend-restart to enable @debug system call family |
[production] |
| 09:02 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:00 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:17 |
<moritzm> |
restarting oozie on an-coord1001 |
[production] |
| 07:54 |
<tarrow@deploy1001> |
Synchronized php-1.34.0-wmf.19/extensions/Wikibase/repo/: Backport for UBN [[gerrit:531527|Hack to avoid trying to termbox render page before save (T230937)]] (duration: 00m 56s) |
[production] |
| 07:46 |
<marostegui> |
Deploy grants on labsdb1009-labsdb1012 to allow connections for haproxy from dbproxy1019 - T202367 |
[production] |
| 06:52 |
<moritzm> |
installing mariadb-10.1 updates from Stretch 9.9 point release (unrelated to wmf-mariadb, mostly client-side clients/libraries as shipped in Debian) |
[production] |
| 06:37 |
<moritzm> |
installing python-pip updates from Stretch 9.9 point release |
[production] |