9501-9550 of 10000 results (80ms)
2019-03-04 §
14:20 <marostegui> Change indexes on logging table on db1100 (s5) and db1097:3314 (commonswiki) - T217397 [production]
14:06 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1097:3314, db1100 to changeindexes on logging tbale (duration: 00m 50s) [production]
13:57 <gehel> restarting blazegraph on wdqs eqiad [production]
12:23 <moritzm> testing component/php72 on mw2224 [production]
11:04 <akosiaris@deploy1001> scap-helm citoid finished [production]
11:04 <akosiaris@deploy1001> scap-helm citoid cluster codfw completed [production]
11:04 <akosiaris@deploy1001> scap-helm citoid upgrade -f citoid-codfw-values.yaml production stable/citoid [namespace: citoid, clusters: codfw] [production]
11:04 <akosiaris@deploy1001> scap-helm citoid cluster eqiad completed [production]
11:04 <akosiaris@deploy1001> scap-helm citoid finished [production]
11:04 <akosiaris@deploy1001> scap-helm citoid upgrade -f citoid-eqiad-values.yaml production stable/citoid [namespace: citoid, clusters: eqiad] [production]
11:04 <akosiaris@deploy1001> scap-helm citoid finished [production]
11:04 <akosiaris@deploy1001> scap-helm citoid cluster staging completed [production]
11:04 <akosiaris@deploy1001> scap-helm citoid upgrade -f citoid-staging-values.yaml staging stable/citoid [namespace: citoid, clusters: staging] [production]
10:53 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More weight to db1089 (duration: 00m 48s) [production]
10:38 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:494191| Bumping portals to master (T128546)]] (duration: 00m 50s) [production]
10:37 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:494191| Bumping portals to master (T128546)]] (duration: 00m 50s) [production]
09:44 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1089 with low weight (duration: 00m 48s) [production]
09:27 <ariel@deploy1001> Finished deploy [dumps/dumps@932bf7e]: make misc dumps failure message nicer (duration: 00m 09s) [production]
09:27 <ariel@deploy1001> Started deploy [dumps/dumps@932bf7e]: make misc dumps failure message nicer [production]
09:22 <godog> temporarily stop prometheus on prometheus2004 to take a snapshot [production]
08:45 <gilles@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T216499 Undo enabling Priority Hints origin trial on ruwiki (duration: 00m 49s) [production]
08:44 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 (duration: 00m 49s) [production]
08:38 <gilles@deploy1001> scap failed: average error rate on 7/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) [production]
08:29 <marostegui> Change logging indexes on db1089 to leave the indexes exactly like the ones on tables.sql - T217397 [production]
08:14 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1089 - T217397 (duration: 00m 49s) [production]
07:48 <ema> cp3032/cp3042: restart varnish-be due to mbox lag [production]
07:42 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 for schema change (duration: 00m 49s) [production]
07:39 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 (duration: 00m 53s) [production]
07:33 <marostegui> Reload haproxy on dbproxy1010 to repool labsdb1010 [production]
07:17 <kart_> Finished manual run of unpublished ContentTranslation draft purge script (T217310) [production]
07:13 <marostegui> Remove dbstore1002 from tendril and zarcillo - T216491 [production]
07:05 <marostegui> Upgrade MySQL on db2088 and db2091 [production]
06:46 <marostegui> Stop MySQL on dbstore1002 for decommission T210478 T172410 T216491 T215589 [production]
06:38 <marostegui> Stop MySQL on labsdb1010 for mysql upgrade [production]
06:34 <gtirloni> downtimed cloudstore1008/9 (T209527) [production]
06:13 <marostegui> Upgrade MySQL on db2041 db2049 db2056 db2095 [production]
06:06 <marostegui> Run analyze table logging on db2038 and db2059 - T71222 [production]
06:05 <marostegui> Reload haproxy on dbproxy1010 to depool labsdb1010 [production]
06:04 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1094:3314 for schema change (duration: 01m 11s) [production]
05:18 <kart_> Started manual run of unpublished ContentTranslation draft purge script (T217310) [production]
2019-03-03 §
12:26 <volans|off> restarted icinga on icinga2001, stale status file, too many open files [production]
10:44 <elukey> restart pdfrender on scb1003 [production]
2019-03-02 §
12:12 <gtirloni> labstore1006 started nfsd T217473 [production]
2019-03-01 §
20:45 <ejegg> turned off fundraising omnimail process unsubscribes job [production]
19:40 <XioNoX> pre-configure asw-a8 ports on asw2-a8-eqiad - T187960 [production]
19:32 <XioNoX> pre-configure asw-a7 ports on asw2-a7-eqiad - T187960 [production]
19:29 <XioNoX> pre-configure asw-a6 ports on asw2-a6-eqiad - T187960 [production]
19:17 <XioNoX> pre-configure asw-a5 ports on asw2-a5-eqiad - T187960 [production]
18:53 <robh> notebook1003 has unusually high load recently (23) and seemed to lag in reporting to icinga. no hardware failures, pinged about it in #wikimedia-analytics [production]
16:33 <jbond42> rolling security update of bind9 packages on jessie and trusty [production]