551-600 of 10000 results (67ms)
2019-07-25 §
12:49 <marostegui> Drop abuse_filter_log.afl_log_id in s4 codfw (lag will appear on codfw) - T226851 [production]
11:53 <marostegui> Compress s3 wikis on labsdb1010 - T222978 [production]
11:03 <arturo> update stretch-wikimedia/thirdparty/kubeadm-k8s on install1002 for T215531 (kubeadm 1.15.1) [production]
10:53 <moritzm> rebooting cloudvirt2003-dev [production]
10:52 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:52 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
10:35 <moritzm> rebooting cloudvirt1024 for kernel update [production]
10:35 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:35 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
09:21 <marostegui> Failover m1 from dbproxy1006 to dbproxy1001 - T227139 [production]
08:54 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:54 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
08:54 <moritzm> rebooting cloudvirt2001-dev [production]
08:32 <Urbanecm> Password reset for SUL user Strejc [production]
08:04 <oblivian@puppetmaster1001> conftool action : set/pooled=yes; selector: cluster=api_appserver,dc=eqiad,name=mw128[0-3].* [production]
08:01 <oblivian@puppetmaster1001> conftool action : set/pooled=yes; selector: cluster=appserver,dc=eqiad,name=mw12(6[89]|7[0-5]).* [production]
08:01 <_joe_> repooling mw1268-1275 in the appserver cluster [production]
08:00 <moritzm> rebooting cloudvirt2001-dev [production]
07:59 <oblivian@puppetmaster1001> conftool action : set/pooled=yes; selector: cluster=api_appserver,dc=eqiad,name=mw12(7[6-9|8[0-3]).* [production]
07:59 <_joe_> repooling mw1276-1283 in the API cluster [production]
07:33 <moritzm> rebooting cloudvirt2001-dev [production]
07:23 <marostegui> Upgrade MySQL on db1072 [production]
07:02 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
07:02 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
06:42 <elukey> restart kafka* on kafka-jumbo1001 to pick up new openjdk-8 version [production]
06:37 <elukey> restart cassandra instances on aqs1004 to pick up new openjdk-8 version [production]
06:34 <elukey> add term eventgate to analytics-in4 on cr1/cr2-eqiad - T228882 [production]
05:31 <twentyafterfour> set phabricator to read-write mode [production]
05:30 <marostegui> Failover m3 from db1072 to db1128 - T228243 [production]
05:30 <twentyafterfour> phabricator set to read-only mode [production]
04:51 <marostegui> Start pre-failover steps on m3 T228243 [production]
02:02 <XioNoX> remove peer AS63541 from cr1-eqsin [production]
2019-07-24 §
23:46 <nuria@deploy1001> Finished deploy [analytics/refinery@7d93398]: deploying refinery 0.0.96 (skipping 0.0.95 due to some jenkins/archiva issues). Try 2 (duration: 13m 34s) [production]
23:43 <catrope@deploy1001> Synchronized php-1.34.0-wmf.15/extensions/Flow: Fix JS error when saving Flow board descriptions (T228818) (duration: 01m 01s) [production]
23:42 <catrope@deploy1001> Synchronized php-1.34.0-wmf.14/extensions/Flow: Fix JS error when saving Flow board descriptions (T228818) (duration: 01m 03s) [production]
23:39 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable homepage for 50% of new users on arwiki (T228120) (duration: 00m 58s) [production]
23:32 <nuria@deploy1001> Started deploy [analytics/refinery@7d93398]: deploying refinery 0.0.96 (skipping 0.0.95 due to some jenkins/archiva issues). Try 2 [production]
23:30 <nuria@deploy1001> Finished deploy [analytics/refinery@834db0a]: deploying refinery 0.0.96 (skipping 0.0.95 due to some jenkins/archiva issues) (duration: 18m 10s) [production]
23:22 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable GrowthExperiments homepage on arwiki (T228120) (duration: 00m 55s) [production]
23:13 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Correct typo in arwiki help panel config (T228820) (duration: 00m 57s) [production]
23:12 <nuria@deploy1001> Started deploy [analytics/refinery@834db0a]: deploying refinery 0.0.96 (skipping 0.0.95 due to some jenkins/archiva issues) [production]
22:41 <thcipriani@> helmfile [EQIAD] Ran 'apply' command on namespace 'blubberoid' for release 'production' . [production]
22:36 <thcipriani@> helmfile [CODFW] Ran 'apply' command on namespace 'blubberoid' for release 'production' . [production]
22:28 <thcipriani@> helmfile [STAGING] Ran 'apply' command on namespace 'blubberoid' for release 'staging' . [production]
21:22 <mutante> <+icinga-wm> RECOVERY - Device not healthy -SMART- on restbase-dev1006 is OK: All metrics within thresholds. (T224260) [production]
21:18 <cscott@deploy1001> Finished deploy [parsoid/deploy@abd05ab]: Updating Parsoid to df1af404 (T227216, T226523, T226451) (duration: 18m 35s) [production]
21:16 <nuria@deploy1001> Finished deploy [analytics/refinery@58e64c1]: deploying refinery 0.0.95 (duration: 03m 54s) [production]
21:12 <nuria@deploy1001> Started deploy [analytics/refinery@58e64c1]: deploying refinery 0.0.95 [production]
21:03 <ppchelko@deploy1001> Finished deploy [restbase/deploy@7911f65]: Store PCS endpoints T222384 (duration: 18m 18s) [production]
21:00 <cscott@deploy1001> Started deploy [parsoid/deploy@abd05ab]: Updating Parsoid to df1af404 (T227216, T226523, T226451) [production]