|
2018-05-15
ยง
|
| 19:53 |
<cwd> |
re-enabled process-control |
[production] |
| 19:42 |
<twentyafterfour@tin> |
scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_770814178" --threads=10 --lang en --quiet' returned non-zero exit status 255 (duration: 02m 47s) |
[production] |
| 19:40 |
<cwd> |
disabled process-control for sql trigger refresh |
[production] |
| 19:39 |
<twentyafterfour@tin> |
Started scap: testwikis wikis to 1.32.0-wmf.4 |
[production] |
| 19:05 |
<mutante> |
mw2139 - wmf-auto-reimage --conftoool --new (because it got "Failed to icinga_downtime" and has a new mainboard (T194426) |
[production] |
| 19:03 |
<mutante> |
mw2139 - wmf-auto-reimage --conftoool --no-verify (T194426) |
[production] |
| 17:27 |
<twentyafterfour> |
branching 1.32.0-wmf.4 refs T191050 |
[production] |
| 17:15 |
<ottomata> |
rolling restart kafka-jumbo100[456] |
[production] |
| 16:36 |
<elukey> |
rolling restart of aqs on aqs* nodes to pick up the new druid config |
[production] |
| 16:27 |
<milimetric@tin> |
Finished deploy [analytics/refinery@679cf09]: Update partition drop script after schema change (duration: 07m 13s) |
[production] |
| 16:25 |
<mobrovac@tin> |
Started restart [cpjobqueue/deploy@58935d5]: (no justification provided) |
[production] |
| 16:21 |
<mobrovac@tin> |
Started restart [changeprop/deploy@e468d8e]: (no justification provided) |
[production] |
| 16:20 |
<milimetric@tin> |
Started deploy [analytics/refinery@679cf09]: Update partition drop script after schema change |
[production] |
| 16:12 |
<elukey> |
roll restart kafka on kafka-jumbo to pick up new zookeeper settings |
[production] |
| 16:11 |
<joal@tin> |
Finished deploy [analytics/aqs/deploy@a736558]: Deploying druid-configuration patch (duration: 05m 47s) |
[production] |
| 16:05 |
<joal@tin> |
Started deploy [analytics/aqs/deploy@a736558]: Deploying druid-configuration patch |
[production] |
| 15:52 |
<elukey> |
rolling restart of hadoop master daemons to pick up new zookeeper settings |
[production] |
| 15:20 |
<elukey> |
roll restart of Kafka Analytics to pick up new zookeeper settings |
[production] |
| 14:59 |
<elukey> |
roll restart of kafka daemons on kafka100[1-3] to pick up new zookeeper settings and group.initial.rebalance.delay.ms = 10s |
[production] |
| 14:28 |
<mobrovac@tin> |
Started restart [changeprop/deploy@e468d8e]: Restart after Kafka settings change |
[production] |
| 14:28 |
<mobrovac@tin> |
Started restart [cpjobqueue/deploy@58935d5]: Restart after Kafka settings change |
[production] |
| 14:19 |
<ottomata> |
temporarily disabling puppet on analytics1003 to run refine-eventbus after jumbo based camus eventbus import finishes |
[production] |
| 14:14 |
<elukey> |
swap conf1001 with conf1004 in the zookeeper main eqiad's config + roll restart of the service |
[production] |
| 14:10 |
<mobrovac@tin> |
Started restart [cpjobqueue/deploy@58935d5]: Restart after Kafka settings change |
[production] |
| 14:09 |
<mobrovac@tin> |
Started restart [changeprop/deploy@e468d8e]: Restart after Kafka settings change |
[production] |
| 14:00 |
<andrewbogott> |
rebooting labnet1001 |
[production] |
| 13:50 |
<elukey> |
roll restart of kafka main codfw (kafka200[1-3]) to pick up group.initial.rebalance.delay.ms = 10s |
[production] |
| 13:31 |
<jynus> |
stop db2055 for reimage |
[production] |
| 13:09 |
<chasemp> |
disable puppet for all openstack things in eqiad |
[production] |
| 13:07 |
<andrewbogott> |
stopping nodepool and puppet on labnodepool1001 for T193579 |
[production] |
| 12:59 |
<andrewbogott> |
stopping puppet on labnet1001 and 1002, silencing icinga for T193579 |
[production] |
| 12:42 |
<jynus> |
stop db2060 for reimage |
[production] |
| 12:14 |
<moritzm> |
uploaded intel-microcode 20180425 for jessie-wikimedia/stretch-wikimedia |
[production] |
| 10:57 |
<jynus> |
stop db2067 for reimage |
[production] |
| 10:49 |
<joal@tin> |
Finished deploy [analytics/refinery@25abeec]: Fix for regular weekly deploy (duration: 06m 45s) |
[production] |
| 10:46 |
<jynus> |
stop db2066 for reimage |
[production] |
| 10:43 |
<joal@tin> |
Started deploy [analytics/refinery@25abeec]: Fix for regular weekly deploy |
[production] |
| 10:16 |
<jynus> |
stop db2065 for reimage |
[production] |
| 10:15 |
<moritzm> |
installing uwsgi security update on graphite servers in eqiad |
[production] |
| 10:07 |
<moritzm> |
installing php5 security updates on trusty |
[production] |
| 09:51 |
<moritzm> |
upgrading API server canaries to HHVM 3,18.5+dfsg-1+wmf8+deb9u1 |
[production] |
| 09:47 |
<jynus> |
stop and restart db2091 for upgrade |
[production] |
| 09:36 |
<joal@tin> |
Finished deploy [analytics/refinery@b2f4c3c]: Regular weekly deploy (duration: 05m 38s) |
[production] |
| 09:30 |
<joal@tin> |
Started deploy [analytics/refinery@b2f4c3c]: Regular weekly deploy |
[production] |
| 09:26 |
<jynus> |
stop and restart db2088 for upgrade |
[production] |
| 09:21 |
<moritzm> |
upgrading app server canaries to HHVM 3,18.5+dfsg-1+wmf8+deb9u1 |
[production] |
| 09:03 |
<jynus> |
stop db2061 for reimage |
[production] |
| 08:42 |
<jynus> |
stop db2068 for reimage |
[production] |
| 03:10 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Tue May 15 03:10:59 UTC 2018 (duration 7m 11s) |
[production] |
| 03:03 |
<l10nupdate@tin> |
scap sync-l10n completed (1.32.0-wmf.3) (duration: 06m 32s) |
[production] |