|
2019-03-06
§
|
| 08:47 |
<marostegui> |
Deploy schema change on s3 codfw, this will generate lag on codfw - T86342 |
[production] |
| 08:42 |
<ema> |
lvs300[12]: upgrade linux to 4.9.144-3.1, reboot for L1TF kernel/microcode updates T203011 |
[production] |
| 08:30 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1090 after MySQL upgrade (duration: 00m 59s) |
[production] |
| 08:15 |
<marostegui> |
Stop MySQL on db1090 for mysql upgrade |
[production] |
| 08:14 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1090 for MySQL upgrade (duration: 00m 56s) |
[production] |
| 08:14 |
<marostegui> |
Reload haproxy on dbproxy1010 to repool labsdb1011 |
[production] |
| 07:48 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Fully repool db1105 after MySQL upgrade (duration: 00m 56s) |
[production] |
| 07:40 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Increase traffic db1105 after MySQL upgrade (duration: 00m 56s) |
[production] |
| 07:34 |
<marostegui> |
Remove dbstore1002 from tendril and zarcillo T216491 |
[production] |
| 07:09 |
<elukey> |
raised analytics user's max_user_connection from 10 to 100 on labsdb1012 - T215231 |
[production] |
| 07:06 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Increase traffic db1105 after MySQL upgrade (duration: 00m 56s) |
[production] |
| 06:53 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Slowly repool db1105 after MySQL upgrade (duration: 00m 56s) |
[production] |
| 06:32 |
<marostegui> |
Stop MySQL on db1105 for MySQL upgrade |
[production] |
| 06:31 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1105 for MySQL upgrade (duration: 01m 14s) |
[production] |
| 06:27 |
<marostegui> |
Add labsdb1012 to tendril and zarcillo - T215231 |
[production] |
| 05:50 |
<kart_> |
Finished manual run of unpublished ContentTranslation draft purge script (T217310) |
[production] |
| 04:26 |
<eileen> |
civicrm revision changed from 196493f372 to 4aac68eead, config revision is 8ca90b4c7b |
[production] |
| 04:00 |
<kart_> |
Started manual run of unpublished ContentTranslation draft purge script (T217310) |
[production] |
| 00:55 |
<twentyafterfour> |
finished US Eveninig SWAT. |
[production] |
| 00:41 |
<twentyafterfour@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: sync https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/494524/ for SWAT refs T217276 (duration: 00m 55s) |
[production] |
| 00:23 |
<twentyafterfour@deploy1001> |
Synchronized wmf-config/mobile.php: sync https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/494271/ for SWAT refs T212253 (duration: 00m 56s) |
[production] |
| 00:12 |
<twentyafterfour@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: sync https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/493236/ for SWAT. refs T217080 (duration: 00m 56s) |
[production] |
|
2019-03-05
§
|
| 23:51 |
<ejegg> |
updated payments-wiki from 4f2935ad17 to f1a89d7045 |
[production] |
| 21:05 |
<godog> |
temporarily stop requests to k8s instance on prometheus2004 |
[production] |
| 21:00 |
<herron> |
restarted apache on grafana1001 |
[production] |
| 20:43 |
<herron> |
retarted apache on grafana1001 |
[production] |
| 19:56 |
<hashar@deploy1001> |
Synchronized php-1.33.0-wmf.20/extensions/LdapAuthentication/: Stop referring to the now-killed AuthPlugin class - T217692 (duration: 00m 57s) |
[production] |
| 17:44 |
<godog> |
bounce uwsgi on graphite1004 |
[production] |
| 17:25 |
<herron> |
restarting uwsgi-graphite-web on graphite1004 |
[production] |
| 16:54 |
<moritzm> |
imported logstash 1:5.6.14-1 to thirdparty/elastic56 |
[production] |
| 16:52 |
<herron> |
restarting uwsgi-graphite-web on graphite1004 |
[production] |
| 16:43 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 16:43 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster staging completed |
[production] |
| 16:43 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade staging stable/eventgate-analytics -f eventgate-analytics-staging-values.yaml [namespace: eventgate-analytics, clusters: staging] |
[production] |
| 16:20 |
<herron> |
restarting uwsgi-graphite-web on graphite1004 |
[production] |
| 15:53 |
<hashar@deploy1001> |
rebuilt and synchronized wikiversions files: group0 to 1.33.0-wmf.20 |
[production] |
| 15:35 |
<hashar@deploy1001> |
Finished scap: testwiki to php-1.33.0-wmf.20 and rebuild l10n cache # T206674 (duration: 51m 03s) |
[production] |
| 14:52 |
<gtirloni> |
reprepro added bdsync_0.10-1+deb9u1 T209527 |
[production] |
| 14:44 |
<hashar@deploy1001> |
Started scap: testwiki to php-1.33.0-wmf.20 and rebuild l10n cache # T206674 |
[production] |
| 14:42 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 14:42 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster eqiad completed |
[production] |
| 14:42 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade production stable/eventgate-analytics -f eventgate-analytics-eqiad-values.yaml [namespace: eventgate-analytics, clusters: eqiad] |
[production] |
| 14:41 |
<otto@deploy1001> |
scap-helm eventgate-analytics finished |
[production] |
| 14:41 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster codfw completed |
[production] |
| 14:41 |
<otto@deploy1001> |
scap-helm eventgate-analytics upgrade production stable/eventgate-analytics -f eventgate-analytics-codfw-values.yaml [namespace: eventgate-analytics, clusters: codfw] |
[production] |
| 14:40 |
<jiji@cumin1001> |
conftool action : set/pooled=yes; selector: dc=codfw,service=citoid,cluster=scb,name=kubernetes.* |
[production] |
| 14:35 |
<hashar@deploy1001> |
scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="cawikibooks" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.BRPBtKvzZH" --verbose' returned non-zero exit status 1 (duration: 00m 20s) |
[production] |
| 14:35 |
<hashar@deploy1001> |
Started scap: testwiki to php-1.33.0-wmf.20 and rebuild l10n cache # T206674 |
[production] |
| 14:34 |
<jijiki> |
Rump up citoid traffic from k8s to 25% on codfw - T213194 |
[production] |
| 14:34 |
<hashar@deploy1001> |
scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="cawikibooks" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.ngh6XIMz8y" --verbose' returned non-zero exit status 1 (duration: 00m 21s) |
[production] |