|
2018-03-07
ยง
|
| 16:08 |
<elukey> |
updating pcc facts for new hosts |
[production] |
| 15:54 |
<moritzm> |
rebooting rdb* fallback hosts in eqiad for kernel security update |
[production] |
| 15:47 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1064, it is not performing well with 2 failed disks - T188685 (duration: 01m 16s) |
[production] |
| 15:26 |
<marostegui> |
Set disk 32:2 on db1064 as offline |
[production] |
| 15:20 |
<moritzm> |
rebooting krypton (running grafana among others) for kernel security update |
[production] |
| 15:17 |
<reedy@tin> |
Synchronized wmf-config/throttle.php: T189121 (duration: 01m 15s) |
[production] |
| 14:45 |
<Amir1> |
EU SWAT is done |
[production] |
| 14:42 |
<ppchelko@tin> |
Finished deploy [cpjobqueue/deploy@aee2eb1]: Increase refreshLinks concurrency to 150 T185052 (duration: 00m 36s) |
[production] |
| 14:41 |
<ppchelko@tin> |
Started deploy [cpjobqueue/deploy@aee2eb1]: Increase refreshLinks concurrency to 150 T185052 |
[production] |
| 14:37 |
<moritzm> |
rebooting rdb* hosts in codfw for kernel security update |
[production] |
| 14:37 |
<ladsgroup@tin> |
Synchronized php-1.31.0-wmf.23/extensions/Wikibase/lib/tests/phpunit/Sites/SiteMatrixParserTest.php: [[gerrit:416745|Add code of special wikis as interwiki when populating sites table, part II (T183019)]] (duration: 01m 16s) |
[production] |
| 14:35 |
<ladsgroup@tin> |
Synchronized php-1.31.0-wmf.23/extensions/Wikibase/lib/includes/Sites/SiteMatrixParser.php: [[gerrit:416745|Add code of special wikis as interwiki when populating sites table (T183019)]] (duration: 01m 16s) |
[production] |
| 14:31 |
<ladsgroup@tin> |
Synchronized php-1.31.0-wmf.24/extensions/Wikibase/lib/tests/phpunit/Sites/SiteMatrixParserTest.php: [[gerrit:416744|Add code of special wikis as interwiki when populating sites table, part II (T183019)]] (duration: 01m 15s) |
[production] |
| 14:27 |
<ladsgroup@tin> |
Synchronized php-1.31.0-wmf.24/extensions/Wikibase/lib/includes/Sites/SiteMatrixParser.php: [[gerrit:416744|Add code of special wikis as interwiki when populating sites table (T183019)]] (duration: 01m 16s) |
[production] |
| 14:19 |
<_joe_> |
adding mwdebug200{1,2} to ganeti in codfw, T187468 |
[production] |
| 14:17 |
<urandom> |
reducing compression chunk length to 32kb on "wikipedia_T_page__summary".data - T189057 |
[production] |
| 14:10 |
<zfilipin@tin> |
Synchronized wmf-config/: SWAT: [[gerrit:416739|Load Wikibase Quality extensions using extension registration (T106104)]] (duration: 01m 17s) |
[production] |
| 14:02 |
<zfilipin@tin> |
Synchronized wmf-config/throttle.php: SWAT: [[gerrit:416585|New throttle rule (T188626)]] (duration: 01m 18s) |
[production] |
| 14:01 |
<urandom> |
setting trace probability to 0.0, restbase eqiad cassandra cluster - T189057 |
[production] |
| 13:22 |
<mobrovac@tin> |
Synchronized wmf-config/InitialiseSettings.php: Switch all refreshLinks jobs to EventBus, file #2 - T185052 (duration: 01m 15s) |
[production] |
| 13:22 |
<moritzm> |
rebooting tungsten for kernel security update |
[production] |
| 13:21 |
<mobrovac@tin> |
Synchronized wmf-config/jobqueue.php: Switch all refreshLinks jobs to EventBus - T185052 (duration: 01m 15s) |
[production] |
| 13:20 |
<ppchelko@tin> |
Finished deploy [cpjobqueue/deploy@d84286a]: Switch all refreshLinks jobs to kafka T185052 (duration: 00m 43s) |
[production] |
| 13:20 |
<moritzm> |
rebooting install2002 for kernel security update |
[production] |
| 13:19 |
<ppchelko@tin> |
Started deploy [cpjobqueue/deploy@d84286a]: Switch all refreshLinks jobs to kafka T185052 |
[production] |
| 10:55 |
<marostegui> |
Deploy schema change on codfw s4 master (db2051) with replication enabled (this will generate lag on codfw) - T187089 T185128 T153182 |
[production] |
| 10:54 |
<moritzm> |
rearmed keyholders on netmon1002 and netmon2001 |
[production] |
| 10:50 |
<elukey> |
reboot stat100[56] for kernel upgrades |
[production] |
| 10:49 |
<moritzm> |
reboot memcached hosts in codfw for kernel security update |
[production] |
| 10:34 |
<moritzm> |
rebooting netmon2001 for kernel security update |
[production] |
| 10:29 |
<moritzm> |
rebooting netmon1002 for kernel security update |
[production] |
| 10:26 |
<moritzm> |
rebooting boron for kernel security update |
[production] |
| 10:11 |
<moritzm> |
rebooting openldap/WMCS servers for kernel security update |
[production] |
| 10:05 |
<moritzm> |
rebooting openldap/corp servers for kernel security update |
[production] |
| 10:03 |
<elukey> |
reboot analytics10[35,52] for kernel updates - hadoop hdfs journal nodes (didn't manage to complete the work yesterday) |
[production] |
| 10:03 |
<moritzm> |
rebooting pool counters in codfw for kernel security update |
[production] |
| 10:02 |
<akosiaris> |
upload apertium-rus-ukr_0.2.0~r82706-1+wmf1 on apt.wikimedia.org/jessie-wikimedia/main. T184901 |
[production] |
| 09:56 |
<moritzm> |
rebooting tureis/roentgenium for kernel security update |
[production] |
| 09:53 |
<akosiaris> |
upload apertium-rus_0.2.0~r82706-1+wmf1 and apertium-ukr_0.1.0~r82563-1+wmf1 on apt.wikimedia.org/jessie-wikimedia/main. T184901 |
[production] |
| 09:46 |
<moritzm> |
rebooting etherpad1001 (etherpad.wikimedia.org) for kernel security update |
[production] |
| 09:31 |
<moritzm> |
rebooting darmstadtium (docker registry) for kernel security update |
[production] |
| 09:24 |
<moritzm> |
rearming keyholder on sarin after reboot |
[production] |
| 09:16 |
<moritzm> |
rebooting sarin for kernel security update |
[production] |
| 08:57 |
<ema> |
cp3033: restart varnish-be, backend connections piling up (~12k) |
[production] |
| 08:40 |
<marostegui> |
Deploy schema change on s7 primary master db1062 - T153182 T185128 |
[production] |
| 08:37 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1079 after alter table (duration: 01m 16s) |
[production] |
| 07:45 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2089,db2079 and db2065 after mariadb and kernel upgrade (duration: 01m 16s) |
[production] |
| 07:30 |
<marostegui> |
Stop mariadb on db2089,db2079 and db2065 for kernel upgrade |
[production] |
| 07:28 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Depool db2089,db2079 and db2065 (duration: 01m 15s) |
[production] |
| 06:49 |
<marostegui> |
Deploy schema change on db1079 with replication enabled (this will generate lag on labs) - T187089 T185128 T153182 |
[production] |