501-550 of 10000 results (57ms)
2019-07-29 §
09:24 <elukey@cumin1001> END (FAIL) - Cookbook sre.druid.roll-restart-workers (exit_code=99) [production]
09:22 <elukey@cumin1001> START - Cookbook sre.druid.roll-restart-workers [production]
09:21 <elukey@cumin1001> END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) [production]
08:55 <elukey@cumin1001> START - Cookbook sre.druid.roll-restart-workers [production]
08:51 <root@> helmfile [STAGING] Ran 'apply' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
08:47 <elukey> set mcrouter async behavior for codfw replication to all mw app/api servers (changes will be picked up when puppet runs on the hosts) - T225642 [production]
08:35 <godog> temp stop puppet on cp hosts to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/525259 [production]
08:32 <elukey@cumin1001> END (ERROR) - Cookbook sre.hadoop.roll-restart-workers (exit_code=97) [production]
08:32 <elukey@cumin1001> START - Cookbook sre.hadoop.roll-restart-workers [production]
08:16 <marostegui> Drop abuse_filter_log.afl_log_id in s7 eqiad - T226851 [production]
07:49 <dcausse> elastic@eqiad force recovery of failed shards (eswiki stuck) [production]
07:30 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db2038 from config T221533 (duration: 00m 46s) [production]
07:29 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db2038 from config T221533 (duration: 00m 50s) [production]
07:18 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) [production]
06:45 <akosiaris> poweroff orespoolcounter{1,2}00{1,2} for removal T227640 [production]
06:37 <_joe_> restarted php7.2 on mwdebug1002, low opcache [production]
06:36 <_joe_> restarted coherence report on netmon1002, it failed earlier this morning [production]
06:31 <_joe_> restarting nrpe on restbase-dev1006 T224260 [production]
06:30 <elukey@cumin1001> START - Cookbook sre.hadoop.roll-restart-workers [production]
05:33 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1104 in preparation for Tuesday 30th failover in s8 (duration: 00m 54s) [production]
05:18 <marostegui> Drop Drop abuse_filter_log.afl_log_id from s7 codfw with replication (this will cause lag in s7 codfw) - T226851 [production]
05:05 <marostegui> Remove db1072 from tendril and zarcillo T228956 [production]
2019-07-28 §
15:13 <arturo> disable 1m load average check in icinga for labstore1007 for 24h [production]
2019-07-27 §
17:39 <bd808> Updated profile & images for @wikimediatech twitter account [production]
14:49 <godog> bounce rsyslog on wezen / centrallog1001 [production]
06:43 <elukey> powercycle mw1300 - no ssh, serial com2 stuck with no root loging available [production]
00:35 <mutante> restbase-dev1006 - starting nagios-nrpe-server [production]
00:33 <mutante> wikitech-static - fix /etc/letsencrypt/renewal/wikitech-static.wikimedia.org.conf - remove webroot_map and and line for status.wm.org that caused errors when doing a renewal dry-run. now dry run finishes succesfully and we are using "webroot" authenticator and not "apache" anymore. This should have resolved what this ticket was about. No more Apache kills/restarts on renewal. (T214640) [production]
2019-07-26 §
23:51 <mutante> restbase-dev1006 - manually booting into PXE to debug boot issue / start Debian installer (T224260) [production]
23:27 <mutante> restbase-dev1006 - does not boot - hangs at "attempting to boot from C:" - entering "Legacy BIOS One Time Boot Menu" (T224260) [production]
21:52 <mutante> restbase-dev1006 - power reset via mgmt [production]
20:48 <mutante> restbase-dev1006 - rebooting from busybox shell where it was idling since a failed reimage attempt [production]
20:22 <foks> reset password for Sharons36 [production]
18:43 <XioNoX> remove lvs100[1-6] switch config from asw2-d-eqiad - T224223 [production]
18:33 <mutante> deploy2001 - delgroup gerrit-root (follow-up to https://gerrit.wikimedia.org/r/c/operations/puppet/+/525444) [production]
18:32 <mutante> deploy1001 - delgroup gerrit-root (follow-up to https://gerrit.wikimedia.org/r/c/operations/puppet/+/525444) [production]
18:20 <XioNoX> remove lvs100[1-6] switch config from asw2-c-eqiad - T224223 [production]
18:08 <XioNoX> remove lvs100[1-6] switch config from asw2-b-eqiad - T224223 [production]
18:01 <XioNoX> remove lvs100[1-6] switch config from asw2-a-eqiad - T224223 [production]
17:37 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
17:37 <robh@cumin1001> START - Cookbook sre.hosts.decommission [production]
16:05 <jforrester@deploy1001> Synchronized php-1.34.0-wmf.15/extensions/Flow/includes/Search/Iterators/TopicIterator.php: T229114 make orderUUID public, as it is needed by other classes for Dumps (duration: 00m 47s) [production]
15:12 <krinkle@deploy1001> Synchronized wmf-config/CommonSettings.php: de0822497919b1b (duration: 00m 48s) [production]
15:02 <Krinkle> krinkle@deploy1001: php-1.34.0-wmf.15 is still dirty on extensions/CheckUser [production]
14:23 <ema> re-enable puppet on cache nodes T229091 [production]
14:10 <ema> disable puppet on cache nodes T229091 [production]
13:41 <fsero> sudo -i reprepro --ignore=wrongdistribution include stretch-wikimedia /home/fsero/envoyproxy_1.11.0~wmf1_amd64.changes [production]
13:41 <jeh> updated labstore100[67].wikimedia.org performance scaling_governor T225713 [production]
13:07 <jeh> rebooting labstore1006.wikimedia.org for updates T224228 [production]
13:00 <Urbanecm> Change user email assigned to SUL user Stansfield (T229004) [production]