2151-2200 of 10000 results (54ms)
2018-07-30 §
11:39 <moritzm> upgrading intel-microcode to 20180703 on the servers which have microcode updates enabled (reboots to pick up the new version will be coordinated separately) [production]
11:33 <tgr@deploy1001> Started scap: T190015 Create separate user group for editing sitewide CSS/JavaScript that does not include administrators by default [production]
11:12 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:449137|Enable reading from change_tag_def everywhere (T199334)]] (duration: 00m 55s) [production]
11:10 <volans> upgrading cumin to 3.0.2 on the remaining cumin masters (neodymium, labpuppetmaster*) [production]
11:01 <volans> upgrading cumin to 3.0.2 on sarin [production]
10:50 <volans> uploaded cumin_3.0.2-1_amd64.deb to apt.wikimedia.org jessie-wikimedia [production]
10:25 <godog> run xfs_repair on sdc1 on ms-be2040 - T199198 [production]
10:04 <akosiaris@deploy1001> scap-helm help finished [production]
10:04 <akosiaris@deploy1001> scap-helm help cluster codfw completed [production]
10:04 <akosiaris@deploy1001> scap-helm help cluster eqiad completed [production]
10:04 <akosiaris@deploy1001> scap-helm help [namespace: help, clusters: eqiad,codfw] [production]
10:03 <akosiaris> upgrade kubernetes staging cluster to 1.9.9 [production]
10:01 <akosiaris> reboot kubestage1001 for kubernetes upgrade [production]
09:59 <akosiaris> reboot kubestage1002 for kubernetes upgrade [production]
09:31 <moritzm> uploaded intel-microcode 20180703 for jessie-wikimedia/stretch-wikimedia to apt.wikimedia.org (tested successfully on a number of canary hosts for approx two weeks) [production]
09:30 <jynus> migrate dbtree to dbmonitor1001 [production]
09:25 <akosiaris@deploy1001> scap-helm mathoid finished [production]
09:14 <akosiaris@deploy1001> scap-helm mathoid cluster codfw completed [production]
09:13 <akosiaris@deploy1001> scap-helm mathoid cluster eqiad completed [production]
09:09 <akosiaris@deploy1001> scap-helm mathoid upgrade production stable/mathoid [namespace: mathoid, clusters: eqiad,codfw] [production]
09:08 <akosiaris> upgrade mathoid helm chart from 0.0.5 to 0.0.9 [production]
09:07 <akosiaris@deploy1001> scap-helm mathoid finished [production]
09:07 <akosiaris@deploy1001> scap-helm mathoid cluster codfw completed [production]
09:07 <akosiaris@deploy1001> scap-helm mathoid cluster eqiad completed [production]
09:07 <akosiaris@deploy1001> scap-helm mathoid upgrade -h [namespace: mathoid, clusters: eqiad,codfw] [production]
08:49 <marostegui> Deploy schema change on dbstore1002:s8 T144010 T51190 T199368 [production]
08:07 <elukey@deploy1001> Finished deploy [eventlogging/analytics@54d43e4]: Band aid for T200630 (duration: 00m 05s) [production]
08:07 <elukey@deploy1001> Started deploy [eventlogging/analytics@54d43e4]: Band aid for T200630 [production]
07:52 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Give some traffic to db1120 to on x1 (duration: 00m 53s) [production]
07:43 <moritzm> installing opencv security updates [production]
07:39 <moritzm> installing mercurial security updates [production]
07:32 <marostegui> Deploy schema change on db2045 (s8 codfw master) this will generate lag on s8 codfw T144010 T51190 T199368 [production]
07:29 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Add db1120 to x1 T196376 (duration: 00m 54s) [production]
07:28 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Add db1120 to x1 T196376 (duration: 00m 59s) [production]
05:28 <marostegui> Deploy schema change on db1068 (commonswiki master) - T51190 [production]
05:00 <marostegui> Deploy schema change on db1062 (s7 primary master) T144010 T51190 T199368 [production]
03:13 <l10nupdate@deploy1001> ResourceLoader cache refresh completed at Mon Jul 30 03:13:42 UTC 2018 (duration 10m 29s) [production]
03:03 <l10nupdate@deploy1001> scap sync-l10n completed (1.32.0-wmf.14) (duration: 15m 33s) [production]
02:28 <l10nupdate@deploy1001> scap sync-l10n completed (1.32.0-wmf.13) (duration: 11m 41s) [production]
2018-07-29 §
09:56 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Depool pc2006 T200641 (duration: 00m 57s) [production]
02:34 <reedy@deploy1001> Synchronized wmf-config/CommonSettings.php: Neuter EducationProgram T188410 (duration: 00m 58s) [production]
2018-07-28 §
17:49 <Krinkle> Navigation Timing alerts indicate we're losing a significant about of traffic [production]
17:35 <elukey> restart eventlogging on eventlog1002 after tons of kafka disconnects (still not clear what happened) [production]
08:16 <jynus@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1094 fully (duration: 00m 57s) [production]
2018-07-27 §
22:05 <ejegg> updated payments-wiki from f0138ab066 to 8c4d6aaa13 [production]
21:39 <ejegg> updated payments-wiki from 5e01f50983 to f0138ab066 [production]
20:40 <SMalyshev> re-pooled wdqs1003 [production]
19:04 <mutante> terbium is being removed from ferm rules on elasticsearch/relforge, logstash/collector, mariadb/labtestwikitech and mw-maintenance itself (T192092) [production]
18:57 <SMalyshev> temporarily depooled wdq1003 to let it catch up with lag [production]
18:15 <mutante> syncing home dirs from mwmaint1001 to mwmaint2001 (once, manually, not currently set to auto-sync, but to keep another copy of former terbium homes in case we failover) (T192092) [production]