|
2018-03-08
§
|
| 10:43 |
<moritzm> |
installing libvpx security updates |
[production] |
| 10:22 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Change db1114 load (duration: 01m 16s) |
[production] |
| 10:14 |
<akosiaris> |
conduct IO stresstests on ganeti1005 (sca1004 VM) with cache=none KVM flag on T181121 |
[production] |
| 10:13 |
<akosiaris> |
conduct IO stresstests on ganeti1005 (sca1004 VM) with cache=none KVM flag on |
[production] |
| 09:57 |
<dcausse> |
restaring mjolnir-kafka-daemon.service on relforge1002 to switch to kafka jumbo |
[production] |
| 09:56 |
<dcausse> |
restaring mjolnir-kafka-daemon.service on relforge1001 to switch to kafka jumbo |
[production] |
| 09:56 |
<_joe_> |
decommissioning mw2017-2099 T187467 |
[production] |
| 09:46 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1114 partially (duration: 01m 16s) |
[production] |
| 09:44 |
<moritzm> |
rearming keyholder on neodymium after reboot |
[production] |
| 09:40 |
<moritzm> |
rebooting neodymium for kernel security update |
[production] |
| 09:22 |
<ema> |
cp-eqsin: reboot for retpoline kernel updates T188092 |
[production] |
| 09:12 |
<ema> |
cp3043: varnish-be-restart T189085 |
[production] |
| 09:08 |
<moritzm> |
rebooting bast1001 for kernel security update |
[production] |
| 08:58 |
<elukey> |
restart varnish backend on cp3041 (failed fetches) |
[production] |
| 08:58 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2046, db2053 and db2060 after kernel upgrade (duration: 01m 15s) |
[production] |
| 08:58 |
<moritzm> |
reset RAC on bast1001, serial console was stuck |
[production] |
| 08:50 |
<elukey> |
rebooting analytics1003 (Hadoop Hive, Oozie, etc..) for kernel updates |
[production] |
| 08:32 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Depool db2046, db2053 and db2060 for kernel upgrade (duration: 01m 17s) |
[production] |
| 08:31 |
<elukey> |
reboot analytics1002 (Hadoop master standby) for kernel upgrades |
[production] |
| 08:28 |
<marostegui> |
Stop MySQL on db2046, db2053 and db2060 for kernel upgrade |
[production] |
| 08:19 |
<elukey> |
reboot analytics1001 (Hadoop master) for kernel upgrade (temp failover to analytics1002) |
[production] |
| 08:09 |
<ema> |
cp3040: varnish-be-restart T189085 |
[production] |
| 08:00 |
<ema> |
cp3032: varnish-be-restart T189085 |
[production] |
| 07:44 |
<elukey> |
reboot kafka2003 (eventbus codfw) for kernel updates |
[production] |
| 07:24 |
<elukey> |
reboot kafka2002 (eventbus codfw) for kernel updates |
[production] |
| 07:22 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool es1019 for maintenance - T187530 (duration: 01m 16s) |
[production] |
| 07:15 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Revert: Depool db1064, it is not performing well with 2 failed disks - T188685 (duration: 01m 31s) |
[production] |
| 04:27 |
<Krinkle> |
Running whisper-mass-resize for ResourceLoader.* metrics on graphite1001 and graphite2001 (T179622) |
[production] |
| 02:30 |
<l10nupdate@tin> |
scap sync-l10n completed (1.31.0-wmf.23) (duration: 07m 37s) |
[production] |
| 02:15 |
<tgr@tin> |
Synchronized wmf-config/throttle.php: T189161 Temporarely remove account creation limit for event on Portuguese Wikipedia on March 08, 2018 (duration: 01m 10s) |
[production] |
| 01:17 |
<twentyafterfour> |
phabricator update completed |
[production] |
| 01:13 |
<twentyafterfour> |
preparing for phabricator update 2018-03-07/1 |
[production] |
| 00:37 |
<thcipriani@tin> |
Synchronized wmf-config/db-eqiad.php: SWAT: [[gerrit:417173|wikitech: use FQDNs for m5 cluster members]] (duration: 01m 16s) |
[production] |
| 00:28 |
<thcipriani@tin> |
Synchronized wmf-config/Wikibase-production.php: SWAT: [[gerrit:413899|Add configuration for CirrusSearch to instantly index new Wikidata items]] T183053 (duration: 01m 15s) |
[production] |
| 00:16 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:416631|Enable loginOnly mode for local auth provider on group 2]] T57420 (duration: 01m 16s) |
[production] |
|
2018-03-07
§
|
| 23:36 |
<MaxSem> |
aborted due to growing DB lag |
[production] |
| 23:08 |
<MaxSem> |
running script for T187516 |
[production] |
| 23:00 |
<maxsem@tin> |
Synchronized php-1.31.0-wmf.24/extensions/AntiSpoof/: https://gerrit.wikimedia.org/r/#/c/417013/ (duration: 01m 16s) |
[production] |
| 22:52 |
<maxsem@tin> |
Synchronized php-1.31.0-wmf.24/extensions/CentralAuth/: https://gerrit.wikimedia.org/r/#/c/417014/ (duration: 01m 20s) |
[production] |
| 22:44 |
<MaxSem> |
dumping centralauth.spoofuser from db1079 |
[production] |
| 22:27 |
<ejegg> |
deployed patch for T171987 to 1.31.0-wmf.23 |
[production] |
| 22:23 |
<ejegg> |
deployed patch for T171987 to 1.31.0-wmf.24 |
[production] |
| 21:51 |
<herron> |
puppetdb server reboots complete — re-enabling puppet agents |
[production] |
| 21:45 |
<herron> |
temporarily disabling puppet agents while puppetdb servers nitrogen and nihal are rebooted for kernel updates |
[production] |
| 21:24 |
<thcipriani@tin> |
Synchronized wmf-config: [[gerrit:416879|Improve load-order documentation for CommonSettings and InitialiseSettings]] noop doc change (duration: 01m 18s) |
[production] |
| 21:05 |
<andrew@tin> |
Synchronized wmf-config/InitialiseSettings.php: Switch wikitech to swift (duration: 01m 15s) |
[production] |
| 20:58 |
<andrew@tin> |
Synchronized wmf-config/filebackend.php: Preparing wikitech to use swift for images, step two (duration: 01m 12s) |
[production] |
| 20:56 |
<andrew@tin> |
Synchronized wmf-config/CommonSettings.php: Preparing wikitech to use swift for images, step one (duration: 01m 16s) |
[production] |
| 20:45 |
<andrew@tin> |
Synchronized multiversion/MWMultiVersion.php: (no justification provided) (duration: 01m 16s) |
[production] |
| 20:27 |
<thcipriani@tin> |
rebuilt and synchronized wikiversions files: Group1 to php-1.31.0-wmf.24 |
[production] |