|
2019-06-25
§
|
| 20:27 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:24 |
<jhuneidi@deploy1001> |
Finished scap: testwikis wikis to 1.34.0-wmf.11 (duration: 41m 35s) |
[production] |
| 20:10 |
<moritzm> |
rebooting webperf hosts for kernel security update |
[production] |
| 20:09 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 20:09 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:00 |
<moritzm> |
rebooting torrelay1001 for kernel security update |
[production] |
| 20:00 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 20:00 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 19:43 |
<jhuneidi@deploy1001> |
Started scap: testwikis wikis to 1.34.0-wmf.11 |
[production] |
| 19:16 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.34.0-wmf.10 refs T220735 |
[production] |
| 19:08 |
<twentyafterfour> |
deploying MediaWiki 1.34.0-wmf.10 to all wikis |
[production] |
| 19:07 |
<twentyafterfour> |
looks like we are unblocked for wmf.10, deploying that first |
[production] |
| 18:45 |
<longma> |
cutting the branch f or 1.34.0-wmf.11 T220736 |
[production] |
| 17:51 |
<jiji@deploy1001> |
Finished deploy [cpjobqueue/deploy@eb8f692]: Migrating RecentChangesUpdate to PHP7 - T219148 (duration: 01m 37s) |
[production] |
| 17:49 |
<jiji@deploy1001> |
Started deploy [cpjobqueue/deploy@eb8f692]: Migrating RecentChangesUpdate to PHP7 - T219148 |
[production] |
| 17:24 |
<smalyshev@deploy1001> |
Finished deploy [wdqs/wdqs@ca96238]: undo: modify agents for T226471 (duration: 16m 14s) |
[production] |
| 17:08 |
<smalyshev@deploy1001> |
Started deploy [wdqs/wdqs@ca96238]: undo: modify agents for T226471 |
[production] |
| 17:01 |
<herron> |
finished migration of kafka2003 to kafka-main2003 — enabling alert notifications for kafka-main2003, and leaving kafka2003 disabled T225005 |
[production] |
| 16:45 |
<reedy@deploy1001> |
Synchronized wmf-config/flaggedrevs.php: Remove some dupe config (duration: 00m 55s) |
[production] |
| 16:25 |
<jynus> |
upgrade and restart db1114 (test-s1) |
[production] |
| 15:59 |
<krinkle@deploy1001> |
Synchronized php-1.34.0-wmf.10/maintenance/: T226448 / 40e725b6502cd6 (duration: 01m 15s) |
[production] |
| 15:56 |
<krinkle@deploy1001> |
Synchronized php-1.34.0-wmf.10/includes/: T226448 / 40e725b6502cd6 (duration: 01m 20s) |
[production] |
| 15:28 |
<jforrester@deploy1001> |
Synchronized php-1.34.0-wmf.10/skins/MonoBook/includes/SkinMonoBook.php: T226503 Fix Notifications RL module dependency (duration: 00m 57s) |
[production] |
| 15:13 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Produce page-properties-change stream to eventgate-main - T211248 (duration: 00m 58s) |
[production] |
| 14:43 |
<herron> |
beginning replacement of kafka2003 with kafka-main2003 T225005 |
[production] |
| 14:26 |
<ottomata> |
shutting down Kafka on old analytics brokers - T183303 |
[production] |
| 14:21 |
<andrewbogott> |
rebooting cloudvirt1014, 1018, 1024 |
[production] |
| 14:02 |
<ema> |
pool cp5002 w/ ATS backend T226477 |
[production] |
| 13:54 |
<onimisionipe> |
changing replication factor of v4 keyspace for maps codfw cluster - T226161 |
[production] |
| 12:49 |
<marostegui> |
Stop MySQL on db1117:m5 (checked dumps, they are done) to clone db1133 - T222682 |
[production] |
| 12:48 |
<godog> |
swift eqiad-prod: put back ms-be1033 - T223518 |
[production] |
| 12:46 |
<ema> |
depool cp5002 and reimage as upload_ats T226477 |
[production] |
| 12:34 |
<jijiki> |
Upgrade scap to eqiad - T224915 |
[production] |
| 12:32 |
<jijiki> |
Upgrade scap to codfw - T224915 |
[production] |
| 12:27 |
<akosiaris> |
fully depool kubernetes2001 T226237 |
[production] |
| 12:26 |
<akosiaris@puppetmaster1001> |
conftool action : set/pooled=inactive; selector: name=kubernetes2001.* |
[production] |
| 12:24 |
<jijiki> |
Upgrade to scap 3.10.0-1 on mw-api-canary as well - T224915 |
[production] |
| 12:22 |
<jijiki> |
Upgrade to scap 3.10.0-1 on mw* codfw |
[production] |
| 10:22 |
<ema> |
pool cp5001 w/ ATS backend T226477 |
[production] |
| 09:30 |
<jijiki> |
enable puppet on dbproxy* |
[production] |
| 09:24 |
<_joe_> |
restarting gerrit on cobalt |
[production] |
| 09:09 |
<ema> |
depool cp5001 and reimage as upload_ats T226477 |
[production] |
| 09:08 |
<jijiki> |
Rolling haproxy restarts on thumbor* - T225284 |
[production] |
| 09:02 |
<jijiki> |
Disable puppet on dbproxy* - T225284 |
[production] |
| 08:51 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Change parsercache key everywhere after deploying it in small batches for a few hours T210725 (duration: 00m 57s) |
[production] |
| 08:50 |
<jijiki> |
Disable puppet on thumbor* - T225284 |
[production] |
| 08:30 |
<marostegui> |
Change parsercachekey on 20 more hosts |
[production] |
| 08:22 |
<smalyshev@deploy1001> |
Finished deploy [wdqs/wdqs@bd3df8c]: modify agents for T226471 (duration: 11m 02s) |
[production] |
| 08:18 |
<marostegui> |
Change parsercachekey on 10 more hosts |
[production] |
| 08:11 |
<smalyshev@deploy1001> |
Started deploy [wdqs/wdqs@bd3df8c]: modify agents for T226471 |
[production] |