|
2018-07-18
§
|
| 10:30 |
<moritzm> |
ran puppet clean/deactivate for lawrencium, long-standing source of errors in package deployments (T191360) |
[production] |
| 10:24 |
<moritzm> |
installing cups/libcups security updates for jessie |
[production] |
| 10:10 |
<dcausse> |
T156137: banning elastic1030 from the cluster (high load) |
[production] |
| 10:02 |
<dcausse> |
clearing internal cache on elastic1030 |
[production] |
| 09:58 |
<hashar> |
contint1001 dropping unused docker containers |
[production] |
| 09:36 |
<moritzm> |
draining restbase1007 for eventual reboot for tests with SSBD microcode |
[production] |
| 08:51 |
<godog> |
run xfs_repair on filesystems reporting negative space available on ms-be1043 - T199198 |
[production] |
| 08:50 |
<moritzm> |
reboot ms-be1040 for tests with SSBD microcode |
[production] |
| 08:45 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1085 after alter table (duration: 00m 53s) |
[production] |
| 08:38 |
<twentyafterfour> |
restart apache on phab1001 |
[production] |
| 08:37 |
<chasemp> |
deploy antivandalism stuff to phab https://gerrit.wikimedia.org/r/c/operations/puppet/+/445329 (needs herald to fully enable) |
[production] |
| 08:31 |
<elukey> |
drain + reboot analytics1030 for kernel updates |
[production] |
| 08:29 |
<marostegui> |
Deploy schema change on db1085 with replication, this will generate lag on labsdb hosts for s6 T144010 T51190 T199368 |
[production] |
| 08:29 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1085 for alter table (duration: 01m 01s) |
[production] |
| 08:19 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1088 after alter table (duration: 00m 53s) |
[production] |
| 08:06 |
<marostegui> |
Deploy schema change on db1088 T144010 T51190 T199368 |
[production] |
| 08:06 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1088 for alter table (duration: 00m 53s) |
[production] |
| 07:41 |
<jforrester@deploy1001> |
Synchronized wmf-config/missing.php: Fix typo in missing.php Ia3aae912d9m (duration: 00m 54s) |
[production] |
| 07:37 |
<moritzm> |
upgrading tor on radium to 0.3.3.9 |
[production] |
| 07:25 |
<marostegui> |
Deploy schema change on db1052 - https://phabricator.wikimedia.org/T191316 https://phabricator.wikimedia.org/T192926 https://phabricator.wikimedia.org/T89737 https://phabricator.wikimedia.org/T195193 |
[production] |
| 07:22 |
<moritzm> |
updated tor on apt.wikimedia.org to 0.3.3.9 |
[production] |
| 06:45 |
<jynus> |
enable semi-sync on db1099:3311 and db1105:3311 |
[production] |
| 06:08 |
<marostegui> |
s1 failover finished T197069 |
[production] |
| 06:07 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: read only OFF after failover T197069 (duration: 00m 53s) |
[production] |
| 06:04 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Set db1067 as master T197069 (duration: 00m 53s) |
[production] |
| 06:01 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Set s1 on ready only for maintenance T197069 (duration: 01m 08s) |
[production] |
| 06:00 |
<marostegui> |
Starting s1 failover from db1052 to db1067 - T197069 |
[production] |
| 05:11 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki dewiki (1 page) - T45917 |
[production] |
| 05:11 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki ruwiki (4 pages) - T45917 |
[production] |
| 05:11 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki frwiki (5 pages) - T45917 |
[production] |
| 05:09 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki metawiki (deleted 1 page) - T45917 |
[production] |
| 05:09 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki test2wiki (deleted 2 pages) - T45917 |
[production] |
| 04:56 |
<marostegui> |
Starting s1 failover pre steps - https://phabricator.wikimedia.org/T197069 |
[production] |
| 03:05 |
<l10nupdate@deploy1001> |
ResourceLoader cache refresh completed at Wed Jul 18 03:05:46 UTC 2018 (duration 10m 27s) |
[production] |
| 02:55 |
<l10nupdate@deploy1001> |
scap sync-l10n completed (1.32.0-wmf.13) (duration: 09m 43s) |
[production] |
| 02:28 |
<l10nupdate@deploy1001> |
scap sync-l10n completed (1.32.0-wmf.12) (duration: 08m 35s) |
[production] |
| 00:00 |
<reedy@deploy1001> |
Finished scap: log event fixes (duration: 33m 18s) |
[production] |
|
2018-07-17
§
|
| 23:26 |
<reedy@deploy1001> |
Started scap: log event fixes |
[production] |
| 22:43 |
<reedy@deploy1001> |
Synchronized wmf-config/interwiki-labs.php: (no justification provided) (duration: 00m 53s) |
[production] |
| 22:24 |
<reedy@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Remove override for foundationwiki in wgMobileUrlTemplate (duration: 00m 53s) |
[production] |
| 22:21 |
<reedy@deploy1001> |
Synchronized tests/urls.txt: testding! (duration: 00m 53s) |
[production] |
| 20:57 |
<reedy@deploy1001> |
Synchronized wmf-config/: make foundation.wikimedia.org writeable T188776 (duration: 00m 53s) |
[production] |
| 20:50 |
<reedy@deploy1001> |
Finished scap: updating l10n cache for foundationwiki url changes (duration: 58m 32s) |
[production] |
| 20:17 |
<ebernhardson> |
un-ban elastic1030 from eqiad search cluster |
[production] |
| 19:52 |
<reedy@deploy1001> |
Started scap: updating l10n cache for foundationwiki url changes |
[production] |
| 19:35 |
<reedy@deploy1001> |
Synchronized portals: T199808 (duration: 00m 54s) |
[production] |
| 19:34 |
<reedy@deploy1001> |
Synchronized portals/wikipedia.org/assets: T199808 (duration: 00m 53s) |
[production] |
| 18:46 |
<bblack> |
re-enabling puppet on mw* for more foundationwiki work - T188776 |
[production] |
| 18:39 |
<ebernhardson> |
ban elastic1030 from eqiad search cluster for latency issues. Likely related to T156137 |
[production] |
| 18:38 |
<reedy@deploy1001> |
Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 02m 53s) |
[production] |