|
2019-08-05
ยง
|
| 15:18 |
<vgutierrez@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=ncredir2002.codfw.wmnet |
[production] |
| 14:31 |
<marostegui> |
Reload haproxy on dbproxy1011 to depool labsdb1010 T222978 |
[production] |
| 14:24 |
<papaul> |
shut down rstbase2009 for battery replacement |
[production] |
| 14:12 |
<fsero@puppetmaster1001> |
conftool action : set/pooled=false; selector: dnsdisc=sessionstore|citoid|cxserver|eventgate-analytics|eventgate-main|termbox|blubberoid|mathoid|zotero,name=codfw |
[production] |
| 14:08 |
<jiji@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 14:08 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:07 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:07 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:06 |
<jijiki> |
Depool and restart restbase2009 for maint - T227408 |
[production] |
| 14:05 |
<fsero@> |
helmfile [CODFW] Ran 'apply' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 14:04 |
<fsero@> |
helmfile [CODFW] Ran 'apply' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' . |
[production] |
| 14:00 |
<fsero@> |
helmfile [CODFW] Ran 'apply' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 13:57 |
<fsero@> |
helmfile [CODFW] Ran 'apply' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 13:56 |
<fsero> |
deploying calico controller in codfw via helmfile - T228837 |
[production] |
| 13:42 |
<fsero> |
deploying tiller in kube-system for helmfile changes - T228837 |
[production] |
| 13:37 |
<volans> |
run cumin 'A:cumin' 'rm -v /usr/local/sbin/{wmf-upgrade-varnish,wmf-upgrade-and-reboot,wmf-downtime-host,wmf-decommission-host}' T205886 |
[production] |
| 13:28 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.kafka.roll-restart-mirror-maker (exit_code=0) |
[production] |
| 13:16 |
<elukey@cumin1001> |
START - Cookbook sre.kafka.roll-restart-mirror-maker |
[production] |
| 13:01 |
<jbond42> |
rolling update of openjdk-8 on restbase |
[production] |
| 12:44 |
<moritzm> |
restarting cassandra on restbase-dev1004 |
[production] |
| 12:44 |
<moritzm> |
restarting cassandra on restbase-dev1040 |
[production] |
| 12:33 |
<moritzm> |
uploaded openjdk-8 u222 for jessie-wikimedia |
[production] |
| 12:26 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki fywiktionary (requested at [[m:Steward_requests/Miscellaneous]]) |
[production] |
| 12:13 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:526657|Switch property terms migration to WRITE_NEW on production wikidata (T225053)]] (duration: 00m 48s) |
[production] |
| 12:01 |
<Urbanecm> |
EU SWAT done |
[production] |
| 11:46 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 0032b0a: Enable Page Previews as default on hewikivoyage (T222017) (duration: 00m 47s) |
[production] |
| 11:43 |
<jbond@cumin1001> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-restart (exit_code=99) |
[production] |
| 11:43 |
<jbond@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-restart |
[production] |
| 11:42 |
<jbond@cumin1001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-restart (exit_code=97) |
[production] |
| 11:42 |
<jbond@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-restart |
[production] |
| 11:38 |
<urbanecm@deploy1001> |
Synchronized php-1.34.0-wmf.16/extensions/MobileFrontend/: SWAT: b7ae4fb: Revert "[AMC] [desktop] [mobile] use AMC by default for desktop users" (T229722) (duration: 00m 49s) |
[production] |
| 11:33 |
<marostegui> |
Upgrade MySQL on db2074 db2057 db2050 db2035 db2098 |
[production] |
| 11:29 |
<urbanecm@deploy1001> |
Synchronized php-1.34.0-wmf.16/extensions/Wikibase: SWAT: 3ecaa57: Add only needed entity usages in AddUsagesForPageJob (T226818, T205045) (duration: 01m 12s) |
[production] |
| 11:04 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 9eb74c2: Define import sources for fawiki (T229717) (duration: 00m 48s) |
[production] |
| 10:40 |
<jbond42> |
update java on sessionstore |
[production] |
| 10:40 |
<jdrewniak@deploy1001> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:528092| Bumping portals to master (T128546)]] (duration: 00m 46s) |
[production] |
| 10:39 |
<jdrewniak@deploy1001> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:528092| Bumping portals to master (T128546)]] (duration: 00m 49s) |
[production] |
| 10:27 |
<ema> |
upload fifo-log-demux 0.5 to stretch-wikimedia |
[production] |
| 10:12 |
<jbond42> |
rolling update of openjdk on maps servers |
[production] |
| 09:30 |
<marostegui> |
Stop MySQL on db2105 to change binlog format |
[production] |
| 09:18 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'restrouter' for release 'staging' . |
[production] |
| 09:07 |
<arturo> |
downtime toolschecker for 5hours |
[production] |
| 09:05 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'restrouter' for release 'staging' . |
[production] |
| 08:56 |
<moritzm> |
installing vim security updates for jessie (stretch/buster already fixed) |
[production] |
| 08:46 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove db2035 from config T229784 (duration: 00m 46s) |
[production] |
| 08:45 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Remove db2035 from config T229784 (duration: 00m 47s) |
[production] |
| 08:43 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'restrouter' for release 'staging' . |
[production] |
| 08:32 |
<marostegui@cumin1001> |
dbctl commit of MediaWiki config (dc=all), diff saved to 'https://phabricator.wikimedia.org/P8861', previous config saved to /var/cache/conftool/dbconfig/20190805-083254-marostegui.json |
[production] |
| 08:21 |
<marostegui> |
Switchover s2 codfw master from db2035 to db2107 - T221533 T220170 |
[production] |
| 07:53 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Reorganize s2 T228969 (duration: 00m 47s) |
[production] |