|
2018-07-18
§
|
| 12:57 |
<elukey> |
restart eventstreams on scb2003 as precautionary after mem consumption too high |
[production] |
| 12:56 |
<elukey> |
restart eventstreams on scb2001 as precautionary after mem consumption too high (still investigating a fix) |
[production] |
| 12:54 |
<dcausse> |
T156137: unbanning elastic1030 |
[production] |
| 12:52 |
<dcausse> |
T156137: restarting elastic1030 to disable G1GC |
[production] |
| 12:51 |
<moritzm> |
installing PHP5 security updates (remaining hosts which only use -cli) |
[production] |
| 12:48 |
<moritzm> |
installing PHP security updates on tungsten |
[production] |
| 12:36 |
<moritzm> |
installing PHP security updates on icinga.wikimedia.org |
[production] |
| 12:24 |
<elukey> |
manually added queued.max.messages.kbytes: 65535 to eventstreams on scb2002 as test for T199813 |
[production] |
| 12:16 |
<moritzm> |
installing PHP security updates on yubiauth servers |
[production] |
| 12:02 |
<moritzm> |
installing PHP security update on bohrium |
[production] |
| 11:35 |
<moritzm> |
installing imagemagick security updates |
[production] |
| 11:34 |
<zeljkof> |
EU SWAT finished |
[production] |
| 11:32 |
<zfilipin@deploy1001> |
Synchronized wmf-config/abusefilter.php: SWAT: [[gerrit:446539|Enable AbuseFilter block option on itwiktionary (T199783)]] (duration: 00m 54s) |
[production] |
| 11:25 |
<moritzm> |
installing policykit security updates on trusty |
[production] |
| 11:20 |
<Amir1> |
making ores tables on euwiki |
[production] |
| 11:15 |
<zfilipin@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:444664|Create Publisher namespace in Bengali Wikisource (T199028)]] (duration: 00m 54s) |
[production] |
| 11:05 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:446563|Enable ores extension wp10 storage in Basque Wikipedia (T198358)]] (duration: 00m 54s) |
[production] |
| 10:30 |
<moritzm> |
ran puppet clean/deactivate for lawrencium, long-standing source of errors in package deployments (T191360) |
[production] |
| 10:24 |
<moritzm> |
installing cups/libcups security updates for jessie |
[production] |
| 10:10 |
<dcausse> |
T156137: banning elastic1030 from the cluster (high load) |
[production] |
| 10:02 |
<dcausse> |
clearing internal cache on elastic1030 |
[production] |
| 09:58 |
<hashar> |
contint1001 dropping unused docker containers |
[production] |
| 09:36 |
<moritzm> |
draining restbase1007 for eventual reboot for tests with SSBD microcode |
[production] |
| 08:51 |
<godog> |
run xfs_repair on filesystems reporting negative space available on ms-be1043 - T199198 |
[production] |
| 08:50 |
<moritzm> |
reboot ms-be1040 for tests with SSBD microcode |
[production] |
| 08:45 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1085 after alter table (duration: 00m 53s) |
[production] |
| 08:38 |
<twentyafterfour> |
restart apache on phab1001 |
[production] |
| 08:37 |
<chasemp> |
deploy antivandalism stuff to phab https://gerrit.wikimedia.org/r/c/operations/puppet/+/445329 (needs herald to fully enable) |
[production] |
| 08:31 |
<elukey> |
drain + reboot analytics1030 for kernel updates |
[production] |
| 08:29 |
<marostegui> |
Deploy schema change on db1085 with replication, this will generate lag on labsdb hosts for s6 T144010 T51190 T199368 |
[production] |
| 08:29 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1085 for alter table (duration: 01m 01s) |
[production] |
| 08:19 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1088 after alter table (duration: 00m 53s) |
[production] |
| 08:06 |
<marostegui> |
Deploy schema change on db1088 T144010 T51190 T199368 |
[production] |
| 08:06 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1088 for alter table (duration: 00m 53s) |
[production] |
| 07:41 |
<jforrester@deploy1001> |
Synchronized wmf-config/missing.php: Fix typo in missing.php Ia3aae912d9m (duration: 00m 54s) |
[production] |
| 07:37 |
<moritzm> |
upgrading tor on radium to 0.3.3.9 |
[production] |
| 07:25 |
<marostegui> |
Deploy schema change on db1052 - https://phabricator.wikimedia.org/T191316 https://phabricator.wikimedia.org/T192926 https://phabricator.wikimedia.org/T89737 https://phabricator.wikimedia.org/T195193 |
[production] |
| 07:22 |
<moritzm> |
updated tor on apt.wikimedia.org to 0.3.3.9 |
[production] |
| 06:45 |
<jynus> |
enable semi-sync on db1099:3311 and db1105:3311 |
[production] |
| 06:08 |
<marostegui> |
s1 failover finished T197069 |
[production] |
| 06:07 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: read only OFF after failover T197069 (duration: 00m 53s) |
[production] |
| 06:04 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Set db1067 as master T197069 (duration: 00m 53s) |
[production] |
| 06:01 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Set s1 on ready only for maintenance T197069 (duration: 01m 08s) |
[production] |
| 06:00 |
<marostegui> |
Starting s1 failover from db1052 to db1067 - T197069 |
[production] |
| 05:11 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki dewiki (1 page) - T45917 |
[production] |
| 05:11 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki ruwiki (4 pages) - T45917 |
[production] |
| 05:11 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki frwiki (5 pages) - T45917 |
[production] |
| 05:09 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki metawiki (deleted 1 page) - T45917 |
[production] |
| 05:09 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki test2wiki (deleted 2 pages) - T45917 |
[production] |
| 04:56 |
<marostegui> |
Starting s1 failover pre steps - https://phabricator.wikimedia.org/T197069 |
[production] |