3751-3800 of 10000 results (83ms)
2019-07-29 §
06:45 <akosiaris> poweroff orespoolcounter{1,2}00{1,2} for removal T227640 [production]
06:37 <_joe_> restarted php7.2 on mwdebug1002, low opcache [production]
06:36 <_joe_> restarted coherence report on netmon1002, it failed earlier this morning [production]
06:31 <_joe_> restarting nrpe on restbase-dev1006 T224260 [production]
06:30 <elukey@cumin1001> START - Cookbook sre.hadoop.roll-restart-workers [production]
05:33 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1104 in preparation for Tuesday 30th failover in s8 (duration: 00m 54s) [production]
05:18 <marostegui> Drop Drop abuse_filter_log.afl_log_id from s7 codfw with replication (this will cause lag in s7 codfw) - T226851 [production]
05:05 <marostegui> Remove db1072 from tendril and zarcillo T228956 [production]
2019-07-28 §
15:13 <arturo> disable 1m load average check in icinga for labstore1007 for 24h [production]
2019-07-27 §
17:39 <bd808> Updated profile & images for @wikimediatech twitter account [production]
14:49 <godog> bounce rsyslog on wezen / centrallog1001 [production]
06:43 <elukey> powercycle mw1300 - no ssh, serial com2 stuck with no root loging available [production]
00:35 <mutante> restbase-dev1006 - starting nagios-nrpe-server [production]
00:33 <mutante> wikitech-static - fix /etc/letsencrypt/renewal/wikitech-static.wikimedia.org.conf - remove webroot_map and and line for status.wm.org that caused errors when doing a renewal dry-run. now dry run finishes succesfully and we are using "webroot" authenticator and not "apache" anymore. This should have resolved what this ticket was about. No more Apache kills/restarts on renewal. (T214640) [production]
2019-07-26 §
23:51 <mutante> restbase-dev1006 - manually booting into PXE to debug boot issue / start Debian installer (T224260) [production]
23:27 <mutante> restbase-dev1006 - does not boot - hangs at "attempting to boot from C:" - entering "Legacy BIOS One Time Boot Menu" (T224260) [production]
21:52 <mutante> restbase-dev1006 - power reset via mgmt [production]
20:48 <mutante> restbase-dev1006 - rebooting from busybox shell where it was idling since a failed reimage attempt [production]
20:22 <foks> reset password for Sharons36 [production]
18:43 <XioNoX> remove lvs100[1-6] switch config from asw2-d-eqiad - T224223 [production]
18:33 <mutante> deploy2001 - delgroup gerrit-root (follow-up to https://gerrit.wikimedia.org/r/c/operations/puppet/+/525444) [production]
18:32 <mutante> deploy1001 - delgroup gerrit-root (follow-up to https://gerrit.wikimedia.org/r/c/operations/puppet/+/525444) [production]
18:20 <XioNoX> remove lvs100[1-6] switch config from asw2-c-eqiad - T224223 [production]
18:08 <XioNoX> remove lvs100[1-6] switch config from asw2-b-eqiad - T224223 [production]
18:01 <XioNoX> remove lvs100[1-6] switch config from asw2-a-eqiad - T224223 [production]
17:37 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
17:37 <robh@cumin1001> START - Cookbook sre.hosts.decommission [production]
16:05 <jforrester@deploy1001> Synchronized php-1.34.0-wmf.15/extensions/Flow/includes/Search/Iterators/TopicIterator.php: T229114 make orderUUID public, as it is needed by other classes for Dumps (duration: 00m 47s) [production]
15:12 <krinkle@deploy1001> Synchronized wmf-config/CommonSettings.php: de0822497919b1b (duration: 00m 48s) [production]
15:02 <Krinkle> krinkle@deploy1001: php-1.34.0-wmf.15 is still dirty on extensions/CheckUser [production]
14:23 <ema> re-enable puppet on cache nodes T229091 [production]
14:10 <ema> disable puppet on cache nodes T229091 [production]
13:41 <fsero> sudo -i reprepro --ignore=wrongdistribution include stretch-wikimedia /home/fsero/envoyproxy_1.11.0~wmf1_amd64.changes [production]
13:41 <jeh> updated labstore100[67].wikimedia.org performance scaling_governor T225713 [production]
13:07 <jeh> rebooting labstore1006.wikimedia.org for updates T224228 [production]
13:00 <Urbanecm> Change user email assigned to SUL user Stansfield (T229004) [production]
12:45 <jeh> rebooting labsdb1012.eqiad.wmnet for updates T224228 [production]
12:35 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Provision db2123 into s5 vslow T221533 (duration: 00m 50s) [production]
09:32 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Provision db2123 into s5 T228969 (duration: 00m 47s) [production]
09:31 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Provision db2123 into s5 T228969 (duration: 00m 48s) [production]
08:42 <marostegui> Add db2123 to tendril and zarcillo - T228969 [production]
06:12 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Fully repool db1096 (duration: 00m 47s) [production]
06:01 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1096 (duration: 00m 47s) [production]
05:45 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to db1096 (duration: 00m 46s) [production]
05:40 <marostegui> Stop MySQL on db1072 to get it ready for decommission - T228956 [production]
05:36 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1096 (duration: 00m 48s) [production]
05:05 <marostegui> Stop MySQL on db1096 for upgrade [production]
05:00 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1096 (duration: 00m 49s) [production]
00:53 <ejegg> re-enabled dedupe_civicrm_contacts and major_gifts_addresses fundraising jobs [production]
00:51 <ejegg> re-enabled donations queue consumer [production]