4651-4700 of 10000 results (55ms)
2019-04-22 §
05:25 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1099 T221502 (duration: 01m 15s) [production]
2019-04-21 §
05:19 <marostegui> Clean up some space on webperf2001 - T221508 [production]
2019-04-20 §
08:12 <_joe_> depooling mw1261,mw1312 wikidata (at least) not working [production]
07:58 <jijiki> Pool thumbor1001 [production]
07:52 <jijiki> depool thumbor1001, switch back to nginx - T187765 [production]
07:50 <_joe_> restarting php-fpm on mw1312, mw1261 to test the new settings over the weekend [production]
2019-04-19 §
23:17 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2245.codfw.wmnet,cluster=api_appserver [production]
23:16 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2244.codfw.wmnet,cluster=api_appserver [production]
23:10 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw2150.codfw.wmnet,service=nginx,cluster=jobrunner [production]
22:55 <mutante> mw2244,mw2245,mw2150 - scap pull [production]
22:53 <mutante> mw2244,mw2245,mw2150 - rebooting for known nutcracker issue after first install [production]
22:47 <mutante> furud - remounted /mnt/hdfs for T221483 [production]
21:42 <mutante> mw2150,mw2244,mw2245: initial puppet run, added to mw roles [production]
19:38 <otto@deploy1001> Synchronized wmf-config/CirrusSearch-common.php: No-op - enabling cirrussearch-request logging in beta (duration: 00m 52s) [production]
19:37 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: No-op - enabling cirrussearch-request logging in beta (duration: 00m 53s) [production]
19:36 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: No-op - prep for enabling cirrussearch-request logging in beta (duration: 00m 53s) [production]
16:20 <bblack> wikipedia.org CNAME TTLs increase to 4H - https://gerrit.wikimedia.org/r/c/operations/dns/+/505249 - T208263 [production]
16:18 <ejegg> rolled back payments-wiki from eb3d0f35de to aa8dad50e7 [production]
15:55 <reedy@deploy1001> Synchronized php-1.34.0-wmf.1/includes/logging/LogFormatter.php: T220767 (duration: 00m 53s) [production]
15:54 <bblack> restart pybal on lvs1016 (eqiad primary) for eventscehmas service add [production]
15:54 <reedy@deploy1001> Synchronized php-1.34.0-wmf.1/includes/Linker.php: T220767 (duration: 00m 55s) [production]
15:50 <bblack@cumin1001> conftool action : set/pooled=yes; selector: name=schema.* [production]
15:42 <bblack> restart pybal on lvs2003 (codfw primary) for eventscehmas service add [production]
15:39 <bblack> restart pybal on lvs2006 (codfw backup) for eventscehmas service add [production]
15:32 <bblack> restarting pybal on lvs1006 (eqiad backup) for eventschema service add [production]
14:59 <volans> uploaded spicerack_0.0.23-1_amd64.deb to apt.wikimedia.org stretch-wikimedia [production]
12:59 <gilles@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T216499 T216598 Enable Priority Hints and Element Timing on eswiki (duration: 00m 56s) [production]
08:45 <akosiaris> restart gerrit to pick up https://gerrit.wikimedia.org/r/504981 [production]
06:39 <elukey> roll restart of druid daemons on druid100[1-3] to pick up new jvm settings [production]
2019-04-18 §
23:16 <mobrovac> evening SWAT completed [production]
23:10 <mobrovac@deploy1001> Synchronized php-1.34.0-wmf.1/extensions/EventBus/includes: (no justification provided) (duration: 00m 54s) [production]
23:10 <ejegg> updated payments-wiki from aa8dad50e7 to eb3d0f35de [production]
23:06 <mobrovac@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Add wikimania years namespaces to wgNamespacesWithSubpages - T220950 (duration: 00m 53s) [production]
23:00 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
23:00 <robh@cumin1001> START - Cookbook sre.hosts.decommission [production]
22:40 <ejegg> updated payments-wiki from aa8dad50e7 to 2f7cd8f195 [production]
22:14 <mutante> LDAP - adding 'ldoan' and 'schang' to 'wmf' (T221118) [production]
22:01 <XioNoX> remove asw2-a-eqiad license keys for troubleshoting [production]
21:58 <ejegg> rolled back payments-wiki to aa8dad50e7 [production]
21:55 <mutante> LDAP - adding rosalie-wmde to group 'wmde' (T220691) [production]
21:52 <ejegg> updated payments-wiki from aa8dad50e7 to 2f7cd8f195 [production]
21:28 <mutante> puppetmaster1001 - mcrouter_generate_certs --generate [production]
21:18 <thcipriani@deploy1001> Finished deploy [gerrit/gerrit@e3c340f]: plugin update -- no restart needed (cobalt) (duration: 00m 10s) [production]
21:18 <thcipriani@deploy1001> Started deploy [gerrit/gerrit@e3c340f]: plugin update -- no restart needed (cobalt) [production]
21:17 <thcipriani@deploy1001> Finished deploy [gerrit/gerrit@e3c340f]: plugin update -- no restart needed (gerrit2001) (duration: 00m 11s) [production]
21:17 <thcipriani@deploy1001> Started deploy [gerrit/gerrit@e3c340f]: plugin update -- no restart needed (gerrit2001) [production]
21:14 <robh@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) [production]
21:14 <robh@cumin1001> START - Cookbook sre.hosts.decommission [production]
20:56 <twentyafterfour@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.34.0-wmf.1 refs T220726 [production]
20:52 <cdanis> root@icinga1001.wikimedia.org /var/lib/icinga # for DOWNTIME in $(fgrep -B12 'comment=mobrovac: temp stop JQ for T221368 - cdanis@cumin1001' retention.dat | grep -A13 servicedowntime | grep downtime_id | cut -d= -f2); do printf "[%lu] DEL_SVC_DOWNTIME;%u\n" $(date +%s) $DOWNTIME ; done > rw/icinga.cmd [production]