|
2019-10-23
§
|
| 06:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1097:3315 for compression T235599', diff saved to https://phabricator.wikimedia.org/P9445 and previous config saved to /var/cache/conftool/dbconfig/20191023-063800-marostegui.json |
[production] |
| 05:29 |
<ema@puppetmaster1001> |
conftool action : set/pooled=false; selector: dnsdisc=kibana,name=codfw |
[production] |
| 05:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1096:3315 after maintenance maintenance', diff saved to https://phabricator.wikimedia.org/P9444 and previous config saved to /var/cache/conftool/dbconfig/20191023-052940-marostegui.json |
[production] |
| 05:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'More traffic to db1096:3315 after maintenance maintenance', diff saved to https://phabricator.wikimedia.org/P9443 and previous config saved to /var/cache/conftool/dbconfig/20191023-050812-marostegui.json |
[production] |
| 04:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'More traffic to db1096:3315 after maintenance maintenance', diff saved to https://phabricator.wikimedia.org/P9442 and previous config saved to /var/cache/conftool/dbconfig/20191023-045722-marostegui.json |
[production] |
| 04:49 |
<vgutierrez> |
repool cp5007 - T234887 |
[production] |
| 04:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1096:3315 after maintenance maintenance', diff saved to https://phabricator.wikimedia.org/P9441 and previous config saved to /var/cache/conftool/dbconfig/20191023-044833-marostegui.json |
[production] |
| 04:36 |
<MaxSem> |
Fixed a page title via namespaceDupes.php on pswiki |
[production] |
| 03:51 |
<vgutierrez> |
depool cp5007 - T234887 |
[production] |
|
2019-10-22
§
|
| 23:57 |
<maxsem@deploy1001> |
Synchronized php-1.35.0-wmf.3/includes/block/DatabaseBlock.php: https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/545373/ (duration: 00m 59s) |
[production] |
| 23:53 |
<maxsem@deploy1001> |
Synchronized wmf-config/wikitech.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/543943/ (duration: 01m 01s) |
[production] |
| 23:43 |
<maxsem@deploy1001> |
Synchronized dblists/: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/543664/ (duration: 00m 59s) |
[production] |
| 23:41 |
<maxsem@deploy1001> |
Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/543664/ (duration: 01m 01s) |
[production] |
| 23:38 |
<maxsem@deploy1001> |
Synchronized dblists/labtestwiki.dblist: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/543664/ (duration: 01m 02s) |
[production] |
| 23:32 |
<mutante> |
LDAP - added keepit-ssh to wmf group (T236209) |
[production] |
| 22:23 |
<ejegg> |
updated Fundraising CiviCRM from ff69d64ad4 to 47e0800001 |
[production] |
| 21:57 |
<thcipriani> |
stopping gerrit to run ref-update script T236114 |
[production] |
| 21:57 |
<thcipriani> |
stopping gerrit to run ref-update script |
[production] |
| 21:45 |
<mutante> |
LDAP - added lexnasser to nda group (T235688) |
[production] |
| 21:07 |
<eileen> |
process-control config revision is 95ee1bafb3 dedupe job re-enabled |
[production] |
| 20:09 |
<mutante> |
gerrit1001 - mkdir /srv/gerrit/cobalt/git - rsyncing /srv/gerrit/git from cobalt to /srv/gerrit/cobalt/git/ on gerrit1001 (T236114) |
[production] |
| 19:42 |
<hashar> |
gerrit1001: apt install colordiff # T236114 |
[production] |
| 19:27 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: group0 to 1.35.0-wmf.3 |
[production] |
| 19:03 |
<brennen> |
proceeding with train for 1.35.0-wmf.3 |
[production] |
| 18:09 |
<mutante> |
DNS - added new Wikipedia language "mnw" (Mon) T235739 - a language spoken in Myanmar |
[production] |
| 17:59 |
<sbassett> |
Uploaded and applied (but did not deploy per releng) security fix for T234450 to wmf.3 |
[production] |
| 17:57 |
<sbassett> |
Deployed security fix for T234450 to wmf.2 |
[production] |
| 17:57 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@b4c484a]: Build structured talk pages by walking the DOM (T235213) (duration: 05m 14s) |
[production] |
| 17:54 |
<mutante> |
restarting gerrit to disable jgit gc (T236114) |
[production] |
| 17:51 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@b4c484a]: Build structured talk pages by walking the DOM (T235213) |
[production] |
| 17:37 |
<arlolra> |
Updated Parsoid to cf01d91 (T234057, T234768, T235296, T235684, T235563) |
[production] |
| 17:26 |
<arlolra@deploy1001> |
Finished deploy [parsoid/deploy@4c64c9c]: Updating Parsoid to cf01d91 (duration: 07m 37s) |
[production] |
| 17:20 |
<bblack> |
geodns: re-pooling esams (at this point, we're entirely back in our "normal" state of affairs) |
[production] |
| 17:19 |
<arlolra@deploy1001> |
Started deploy [parsoid/deploy@4c64c9c]: Updating Parsoid to cf01d91 |
[production] |
| 16:51 |
<bblack> |
geodns: moving all "normal" eqiad traffic back to eqiad (in addition to the esams-diverted traffic which is still pointed mostly at eqiad right now) |
[production] |
| 16:21 |
<mutante> |
running puppet on deployment servers |
[production] |
| 16:20 |
<thcipriani> |
restarting gerrit |
[production] |
| 16:14 |
<thcipriani> |
stopping gerrit to run a fix for T222391 |
[production] |
| 15:58 |
<bblack> |
depooling esams temporarily to test traffic scenario on lvs1014 |
[production] |
| 15:47 |
<bblack> |
enable pybal+puppet on rebooted lvs1014 |
[production] |
| 15:40 |
<bblack> |
rebooting lvs1014 |
[production] |
| 15:28 |
<liw@deploy1001> |
Finished scap: testwiki to php-1.35.0-wmf.3 and rebuild l10n cache (duration: 37m 39s) |
[production] |
| 15:26 |
<XioNoX> |
repool esams |
[production] |
| 15:20 |
<XioNoX> |
rollback ns2 redirect |
[production] |
| 15:13 |
<bblack> |
re-disabling lvs1014 ... |
[production] |
| 15:10 |
<bblack> |
re-enabling lvs1014 pybal/puppet |
[production] |
| 15:03 |
<moritzm> |
rebooting kafka-main1005 for microcode debugging |
[production] |
| 15:01 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:01 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:52 |
<bblack> |
stopping puppet and pybal on lvs1014 (upload+maps traffic to 1016) |
[production] |