|
2019-07-16
ยง
|
| 22:28 |
<fsero> |
depooling ms-fe2005 for swift upload for registry T228196 |
[production] |
| 22:26 |
<mutante> |
wikitech-static ran certbot with --dry-run renew to confirm cert renewal works and it was just fine .. 2 minutes later apache errors which were fixed by restarting apache2 (T214640) |
[production] |
| 22:24 |
<mutante> |
wikitech-static restarted apache |
[production] |
| 22:11 |
<mutante> |
wikitech-static: turn /etc/apache2/sites-available/wikitech-static.wikimedia.org-ssl.conf and status.wikimedia.org-ssl.conf into symlinks to /wikitech-static/apache/ to match config for http vhosts (T225258) |
[production] |
| 22:06 |
<mutante> |
wikitech-static: move /etc/apache2/sites-available/000-default.conf and default-ssl.conf out of directory and reload apache to confirm they are not used and get us in sync with the repo contents again (T225258) |
[production] |
| 21:58 |
<hauskatze> |
GitHub: deleted `wikimedia/mediawiki-extensions-Convert2Wiki` refs. T228198 |
[releng] |
| 21:29 |
<James_F> |
Docker: Publishing a whole new set of RelEng images for T228196 |
[releng] |
| 21:17 |
<bd808@deploy1001> |
Finished deploy [striker/deploy@247a8a6]: Fixes for ssh key management, git repo creation, and Django upgrade (T221657, T227508) (duration: 01m 08s) |
[production] |
| 21:15 |
<bd808@deploy1001> |
Started deploy [striker/deploy@247a8a6]: Fixes for ssh key management, git repo creation, and Django upgrade (T221657, T227508) |
[production] |
| 20:55 |
<SMalyshev> |
repooled wdqs2004 and wdqs2001 - reload done |
[production] |
| 20:26 |
<mutante> |
ganeti1001 - gnt-instance remove netmon1003.wikimedia.org (T220355) |
[production] |
| 20:25 |
<James_F> |
Docker: Running a general rebuild for all missing RelEng images T228196 |
[releng] |
| 19:59 |
<XioNoX> |
update ACLs on pfw3-eqiad/codfw - T228205 |
[production] |
| 19:52 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
| 19:51 |
<fsero> |
republishing base images for wikimedia-(stretch,jessie and buster) T228196 |
[production] |
| 18:58 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 18:58 |
<gehel@cumin1001> |
END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) |
[production] |
| 18:58 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 18:54 |
<gehel> |
data copy from wdqs2004 to wdqs2001 - T228122 |
[production] |
| 18:46 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: retry - Produce revision-create stream to eventgate-main - T211248 (duration: 00m 54s) |
[production] |
| 18:22 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Produce revision-create stream to eventgate-main - T211248 (duration: 00m 54s) |
[production] |
| 18:08 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Update ExtensionDistributor config to point to REL1_33 as the released version (duration: 00m 54s) |
[production] |
| 18:05 |
<fsero> |
republishing base images for nodejs-slim due to registry T228196 |
[production] |
| 18:02 |
<andrewbogott> |
rebooting cloudcontrol2003-dev, cloudweb2001-dev, cloudcontrol1004 for T225713 |
[production] |
| 17:39 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Produce centralnotice.campaign-* streams to eventgate-main - T211248 (duration: 00m 55s) |
[production] |
| 17:23 |
<bsitzmann@deploy1001> |
Finished deploy [mobileapps/deploy@cb6e7bc]: Update mobileapps to 334a4c4 (T227907) (duration: 04m 51s) |
[production] |
| 17:19 |
<bsitzmann@deploy1001> |
Started deploy [mobileapps/deploy@cb6e7bc]: Update mobileapps to 334a4c4 (T227907) |
[production] |
| 16:55 |
<mutante> |
netmon1003: shutdown -h now | ganeti1001: gnt-instance shutdown netmon1003.wikmedia.org - removed from icinga T198939 T220355 |
[production] |
| 16:49 |
<James_F> |
Zuul: [ContentTranslation] Adding CentralAuth to phan dependencies |
[releng] |
| 16:48 |
<James_F> |
Zuul: Adding first tasks for mediawiki/services/kartotherian T228170 |
[releng] |
| 16:36 |
<jiji@deploy1001> |
Finished deploy [cpjobqueue/deploy@5d8128e]: Migrating videoscaling jobs to PHP7 - T219150 (duration: 00m 50s) |
[production] |
| 16:35 |
<jiji@deploy1001> |
Started deploy [cpjobqueue/deploy@5d8128e]: Migrating videoscaling jobs to PHP7 - T219150 |
[production] |
| 16:28 |
<dcausse> |
reindexing wikidata (elastic@eqiad) T227136 |
[production] |
| 15:57 |
<tarrow@> |
helmfile [STAGING] Ran 'apply' command on namespace 'termbox' for release 'staging' . |
[production] |
| 15:37 |
<elukey> |
reboot analytics1072 as attempt to force the raid controller to set a drive failed - T226467 |
[production] |
| 15:19 |
<James_F> |
Ran manual scap pull on deployment-deploy01 |
[releng] |
| 15:12 |
<elukey> |
start mariadb on db1107 and re-enable mysql consumers on eventlog1002 and replication on db1108 |
[production] |
| 15:10 |
<James_F> |
beta-scap-eqiad failing because mw-config repo is out of date? |
[releng] |
| 14:57 |
<James_F> |
Taking deployment-deploy01 offline |
[releng] |
| 14:53 |
<elukey> |
stop mariadb on db1107 to allow maintenance |
[production] |
| 14:53 |
<elukey> |
stop eventlogging mysql consumers on eventlog1002 and eventlogging_sync on db1108 to allow db1107 maintenance |
[production] |
| 14:52 |
<jbond42> |
will restart redis on oresdb at 16:00 UTC - T228045 |
[production] |
| 14:51 |
<jbond42> |
enable puppet accross the fleat |
[production] |
| 14:50 |
<James_F> |
Jenkins deadlock? Beta jenkins jobs haven't successfully run for ~20 hours. |
[releng] |
| 14:50 |
<jbond@cumin1001> |
conftool action : set/pooled=yes; selector: name=dns1001.wikimedia.org |
[production] |
| 14:43 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:43 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:43 |
<jbond@cumin1001> |
conftool action : set/pooled=no; selector: name=dns1001.wikimedia.org |
[production] |
| 14:40 |
<jbond42> |
disable puppet accross the fleat to make a change to the hiera |
[production] |
| 14:29 |
<jijiki> |
Enable puppet and rolling restart thumbor* in codfw - T224572 |
[production] |