8751-8800 of 10000 results (50ms)
2017-12-04 §
14:30 <elukey> reboot druid100[23] for kernel updates [production]
14:28 <zfilipin@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:394771|Localize sitename and meta NS for wawiktionary (T181782)]] (duration: 00m 46s) [production]
14:01 <elukey> reboot analytics106* (hadoop worker nodes) for kernel+jvm updates - T179943 [production]
13:24 <paravoid> upgrading bast2001 to stretch [production]
13:22 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1105:3312 - T174569 (duration: 00m 45s) [production]
13:20 <marostegui> Deploy schema change on db1105:3312 (s2) - T174569 [production]
13:05 <marostegui> Compress s6 on db1098 - T178359 [production]
12:44 <gehel@tin> Finished deploy [kartotherian/deploy@e166d87]: testing new kartotherian packaging on maps-test2003 (duration: 00m 22s) [production]
12:43 <gehel@tin> Started deploy [kartotherian/deploy@e166d87]: testing new kartotherian packaging on maps-test2003 [production]
11:06 <jdrewniak@tin> Synchronized portals: Wikimedia Portals Update: [[gerrit:394964|Bumping portals to master (T128546)]] (duration: 00m 45s) [production]
11:05 <jdrewniak@tin> Synchronized portals/prod/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:394964|Bumping portals to master (T128546)]] (duration: 00m 45s) [production]
10:37 <jynus@tin> Synchronized wmf-config/db-codfw.php: Repool db2085 (duration: 00m 43s) [production]
10:06 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Fully pool db1096:3316 - T178359wq! (duration: 00m 45s) [production]
09:55 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db1099:3318 from s5 (duration: 00m 44s) [production]
09:51 <godog> bootstrap restbase1012-c - T179422 [production]
09:32 <godog> clear erroneous table metrics from graphite1003 / graphite2002 - T181689 [production]
09:24 <elukey> reboot analytics104* (hadoop worker nodes) for kernel+jvm updates - T179943 [production]
09:19 <jynus> rebooting mariadb at labsdb1005 [production]
09:12 <moritzm> reimaging mw1259 (video scaler) to stretch, will be kept disabled initially (some controlled live tests following) [production]
08:57 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1096:3315 and 3316 - T178359 (duration: 00m 45s) [production]
08:45 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1096:3316 - T178359 (duration: 00m 45s) [production]
08:44 <moritzm> updating tor on radium to 0.3.1.9 [production]
08:41 <moritzm> updating tor packages to 0.3.1.9 [production]
08:30 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1096:3315 and pool db1096:3316 - T178359 (duration: 00m 45s) [production]
08:12 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Pool db1096:3315 - T178359 (duration: 00m 44s) [production]
08:11 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Pool db1096:3315 - T178359 (duration: 00m 45s) [production]
07:53 <moritzm> installing curl security updates [production]
07:17 <marostegui> Compress s1 on db1099 - T178359 [production]
07:08 <marostegui> Stop MySQL on db1044 as it will be decommissioned - T181696 [production]
07:05 <_joe_> playing with puppetdb status for ores2003 (deactivating/reactivating node) [production]
06:40 <marostegui> Stop MySQL on db1098 to clone db1096.s6 - T178359 [production]
06:39 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Remove db1044 from config as it will be decommissioned - T181696 (duration: 00m 45s) [production]
06:38 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db1044 from config as it will be decommissioned - T181696 (duration: 00m 45s) [production]
06:34 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1098 - T178359 (duration: 00m 46s) [production]
06:21 <marostegui> Deploy alter table on s3 master (db1075) without replication - T174569 [production]
02:32 <l10nupdate@tin> scap sync-l10n completed (1.31.0-wmf.10) (duration: 06m 28s) [production]
2017-12-03 §
15:33 <ejegg> disabled CiviCRM bounce processing job [production]
12:17 <akosiaris> empty ganeti1006, it had issues this morning per T181121 [production]
12:06 <marostegui> Fix dbstore1002 replication [production]
07:44 <akosiaris> ran puppet on conf2002, etcdmirror-conftool-eqiad-wmnet got started again [production]
05:11 <andrewbogott> deleting files on labsdb1003 /srv/tmp older than 30 days [production]
03:57 <no_justification> gerrit2001: icinga is flapping on the gerrit process/systemd check, but this is kind of known (not sure why it's doing this all of a sudden). It's not letting me acknowledge it, but it's fine/harmless. Cf T176532 [production]
2017-12-02 §
17:55 <marostegui> Reboot db1096.s5 to pick up the correct innodb_buffer_pool size after finishing compressing s5 - T178359 [production]
03:51 <hoo> Ran "scap pull" on snapshot1001, after final T181385 tests [production]
00:03 <mutante> tried one more time on db2028,db2029, both trusty. on db2028: gmond was running as user ganglia-monitor, failed, had to manually kill the process, run puppet again then ok. on db2029, gmond was running as "499" but puppet just ran and removed it without manual intervention. (T177225) [production]
2017-12-01 §
23:15 <urandom> starting cassandra bootstrap, restbase1012-b - T179422 [production]
21:49 <mutante> db2029 - removing ganglia-monitor, testing to kill gmond, running puppet to figure out how to cleanly remove it on trusty [production]
21:12 <mutante> db2023 killed gmond (ganglia-monitor) process manually which was still running even though ganglia-monitor package was removed and caused puppet breakage (it seems only on trusty). after that puppet run is clean again and ganglia removed. (T177225) (https://gerrit.wikimedia.org/r/#/c/394647/1) [production]
20:18 <awight@tin> Started deploy [ores/deploy@9afbf14]: (non-production) Test ORES deployment to ores100* [production]
20:17 <awight@tin> Finished deploy [ores/deploy@9afbf14]: (non-production) Test ORES deployment to ores1001 (duration: 02m 31s) [production]