|
2019-05-09
§
|
| 12:01 |
<_joe_> |
reenabling puppet across the fleet |
[production] |
| 11:57 |
<jbond42> |
all puppetmasters and puppetdbs should be restored' |
[production] |
| 11:55 |
<jbond42> |
clean up old source files sudo cumin A:puppetmaster 'rm /etc/apt/sources.list.d/component-facter3.list /etc/apt/sources.list.d/component-puppet5.list' |
[production] |
| 11:49 |
<volans> |
updated netbox statues for decommissioning and spare hosts according to T222352 |
[production] |
| 11:23 |
<jbond42> |
running sudo apt-get install puppet-master=4.8.2-5~bpo8+1 puppet-master-passenger=4.8.2-5~bpo8+1 on labtestpuppetmaster2001 |
[production] |
| 11:19 |
<jbond42> |
running sudo apt-get install facter=2.4.6-1 puppet=4.8.2-5 puppet-master puppet-master-passenger on labpuppetmaster1001 |
[production] |
| 11:18 |
<jbond42> |
starting puppetdb on puppetdb2001 |
[production] |
| 11:15 |
<jbond42> |
run sudo apt-get install puppetdb on puppetdb2001 |
[production] |
| 11:14 |
<jbond42> |
ran the folloowing on puppetdb2001 sudo apt-get install facter=2.4.6-1 puppet=4.8.2-5 |
[production] |
| 11:14 |
<jbond42> |
ran the folloowing on puppetmaster200{1,2} sudo apt-get install facter=2.4.6-1 puppet=4.8.2-5 puppet-master puppet-master-passenger |
[production] |
| 11:04 |
<_joe_> |
disabling puppet across the fleet |
[production] |
| 11:02 |
<volans> |
stopped ircecho to avoid spam |
[production] |
| 10:43 |
<marostegui> |
Stop MySQL on db1081 |
[production] |
| 10:41 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1081 (duration: 00m 57s) |
[production] |
| 10:29 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Give API traffic to db1129 (new host on s2) (duration: 00m 57s) |
[production] |
| 10:15 |
<_joe_> |
restarting low-traffic pybals in codfw, eqiad |
[production] |
| 10:05 |
<akosiaris> |
restart proton on proton1001. Host Out of memory T214975 |
[production] |
| 09:57 |
<ariel@deploy1001> |
Finished deploy [dumps/dumps@ab56fdd]: reduce sleep time between dumps of adds-changes wikis still more (retry) (duration: 00m 06s) |
[production] |
| 09:57 |
<ariel@deploy1001> |
Started deploy [dumps/dumps@ab56fdd]: reduce sleep time between dumps of adds-changes wikis still more (retry) |
[production] |
| 09:54 |
<ariel@deploy1001> |
Finished deploy [dumps/dumps@ab56fdd]: reduce sleep time between dumps of adds-changes wikis still more (duration: 00m 06s) |
[production] |
| 09:54 |
<ariel@deploy1001> |
Started deploy [dumps/dumps@ab56fdd]: reduce sleep time between dumps of adds-changes wikis still more |
[production] |
| 09:31 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: More traffic to db1129 (new host on s2) (duration: 00m 57s) |
[production] |
| 09:29 |
<fsero@puppetmaster1001> |
conftool action : set/pooled=true; selector: dnsdisc=docker-registry,name=codfw |
[production] |
| 09:12 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: More traffic to db1076 and db1129 (new host on s2) (duration: 00m 56s) |
[production] |
| 09:12 |
<godog> |
bounce rsyslog on lithium |
[production] |
| 09:01 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: More traffic to db1076 and db1129 (new host on s2) (duration: 00m 56s) |
[production] |
| 08:49 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: More traffic to db1076 and db1129 (new host on s2) (duration: 00m 56s) |
[production] |
| 08:38 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: More traffic to db1076 and db1129 (new host on s2) (duration: 00m 57s) |
[production] |
| 08:28 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: More traffic to db1076 (duration: 00m 55s) |
[production] |
| 08:23 |
<elukey> |
upload uwsgi 2.0.14+20161117-3+deb9u2+wmf1 packages to stretch-wikimedia - T212697 |
[production] |
| 08:21 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Pool db1129 with low weight on s2 - T222682 (duration: 00m 56s) |
[production] |
| 08:13 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Slowly repool db1076 (duration: 00m 56s) |
[production] |
| 08:07 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Provision db1129, db2104, db2107, db2108 T222772 T222682 (duration: 00m 57s) |
[production] |
| 08:06 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Provision db1129, db2104, db2107, db2108 T222772 T222682 (duration: 00m 59s) |
[production] |
| 07:54 |
<moritzm> |
installing jquery security updates for stretch |
[production] |
| 07:50 |
<elukey> |
roll restart HDFS masters on an-master100[1,2] to pick up new logging settings |
[production] |
| 07:23 |
<moritzm> |
installing twitter-bootstrap3 security updates |
[production] |
| 06:53 |
<_joe_> |
restarted nagios-nrpe-server on proton1001 |
[production] |
| 05:58 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Clarify disk status for db2103, db2112, db2116 (duration: 00m 58s) |
[production] |
| 05:29 |
<marostegui> |
Stop replication on db2098:s2 |
[production] |
| 05:25 |
<marostegui> |
Stop MySQL on db1076 |
[production] |
| 05:22 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1076 (duration: 00m 57s) |
[production] |
| 05:09 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Pool db2103, db2112 and db2116 into s1 T222772 (duration: 01m 41s) |
[production] |
| 05:07 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Pool db2103, db2112 and db2116 into s1 T222772 (duration: 01m 22s) |
[production] |
| 04:55 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool pc1007 (duration: 00m 59s) |
[production] |
| 00:57 |
<twentyafterfour> |
stopped phd, now running `puppet agent --test` manually on phab1001 |
[production] |
| 00:08 |
<twentyafterfour> |
phabricator upgrade successful |
[production] |
| 00:04 |
<twentyafterfour> |
starting phabricator deployment, momentary downtime expected (~1 minute) |
[production] |