501-550 of 10000 results (62ms)
2019-02-11 §
09:38 <marostegui> Stop all mysql instances on dbstore1005 for reboot [production]
09:11 <marostegui> Stop all mysql instances on dbstore1003 for reboot [production]
08:17 <moritzm> removed cloudcontrol2001-dev.codfw.wmnet from debmonitor (actual hostname in use is cloudcontrol2001-dev.wikimedia.org) [production]
08:07 <marostegui> Deploy schema change on s8 codfw master (db2045) - this will generate lag on codfw T210713 [production]
07:43 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Fully repool db1100 (duration: 00m 46s) [production]
07:39 <marostegui> Deploy schema change on s7 primary master (db1062) - T210713 [production]
07:27 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Give api traffic to db1100 (duration: 00m 46s) [production]
07:18 <marostegui> Stop all mysql instances on dbstore1004 for a reboot [production]
07:17 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1100 with low weight (duration: 00m 46s) [production]
07:06 <marostegui> Upgrade MySQL on db1100 [production]
07:06 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1100 for mysql upgrade (duration: 00m 47s) [production]
07:00 <marostegui> Restart icinga on icinga1001 - checks went awol [production]
06:51 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1079 (duration: 00m 48s) [production]
06:14 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1079 (duration: 00m 48s) [production]
06:14 <marostegui@deploy1001> sync-file aborted: Depool db0179 (duration: 00m 01s) [production]
04:23 <TimStarling> on mwmaint1002: running normalizeThrottleParameters.php --dry-run on all wikis (T209565) [production]
04:19 <tstarling@deploy1001> Synchronized php-1.33.0-wmf.16/extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php: maintenance script update for new dry run (duration: 00m 47s) [production]
04:19 <tstarling@deploy1001> Synchronized php-1.33.0-wmf.16/extensions/WikimediaEvents/tests/phpunit/PageViewsTest.php: test-only undeployed change (duration: 00m 46s) [production]
04:18 <tstarling@deploy1001> Synchronized php-1.33.0-wmf.16/extensions/NavigationTiming/tests/ext.navigationTiming.test.js: test-only undeployed change (duration: 00m 51s) [production]
04:10 <tstarling@deploy1001> sync-file aborted: test-only undeployed change (duration: 00m 12s) [production]
03:05 <kartik@deploy1001> Finished deploy [cxserver/deploy@ee4a15a]: Update cxserver to 8928852 (T213256) (duration: 04m 08s) [production]
03:00 <kartik@deploy1001> Started deploy [cxserver/deploy@ee4a15a]: Update cxserver to 8928852 (T213256) [production]
2019-02-10 §
19:59 <volans|off> force rebooting mw1299, stuck again - T215569 [production]
19:16 <volans|off> forcing reboot of icinga1001 because it's stuck again (no ping, no ssh, CPU stuck messages on console) - T214760 [production]
09:25 <marostegui> Disable notifications for lag checks on dbstore1002 - T210478 [production]
2019-02-09 §
21:42 <Reedy> running `foreachwiki refreshImageMetadata.php --mediatype BITMAP --mime image/vnd.djvu --force` on mwmaint1002 T215635 [production]
21:41 <Reedy> refreshImageMetadata.php for commonswiki done T215635 [production]
16:51 <Jeff_Green> restarted icinga process on icinga1001 because of passive check alert-storm [production]
2019-02-08 §
23:23 <Reedy> running `refreshImageMetadata.php --mediatype BITMAP --mime image/vnd.djvu --force` against commonswiki on mwmaint1002 T215635 (this time we mean it) [production]
22:56 <Reedy> running `refreshImageMetadata.php --mediatype BITMAP --mime image/vnd.djvu` against commonswiki on mwmaint1002 T215635 [production]
21:25 <reedy@deploy1001> Synchronized multiversion/MWMultiVersion.php: Move variable (duration: 00m 49s) [production]
19:50 <krinkle@deploy1001> Synchronized w/touch.php: Ia1e610a5f (duration: 00m 46s) [production]
19:49 <krinkle@deploy1001> Synchronized w/robots.php: Ia1e610a5f (duration: 00m 46s) [production]
19:48 <krinkle@deploy1001> Synchronized w/favicon.php: Ia1e610a5f (duration: 00m 46s) [production]
19:47 <krinkle@deploy1001> Synchronized w/extract2.php: Ia1e610a5f (duration: 00m 48s) [production]
18:14 <gtirloni> T213527 graphite2002 disabled puppet and commented prometheus_puppet_agent_stats cronjob due to cronspam [production]
18:08 <jynus@deploy1001> Synchronized wmf-config/db-eqiad.php: Increase weight for s1 rc slaves (duration: 00m 49s) [production]
17:54 <mutante> phab1001 - restart aphlict service [production]
17:52 <mutante> phab1001 - restarting phd service [production]
17:49 <arturo> T215605 add prometheus-openstack-exporter 0.0.8-4 to stretch-wikimedia [production]
17:47 <mutante> phab1001 - restarting apache2 service for library upgrade [production]
17:42 <mutante> graceful reload of apache on phabricator prod server (phab1001) [production]
17:27 <XioNoX> merge Icinga: add ping check for ulsfo PDUs [production]
16:50 <ejegg> updated payments-wiki-staging from 52a271e681 to 31647bc97e [production]
16:09 <jynus> stopping s1 replication on dbstore1001 to speed up cloning T214720 [production]
16:08 <moritzm> imported git-fat 0.1.3-2+deb10u1 to buster-wikimedia (T213527) [production]
15:46 <marostegui> Repool labsdb1009 - T212308 [production]
15:33 <_joe_> apt-get upgrade on mwmaint2001 to fix the php installation T215376 [production]
15:31 <moritzm> imported debmonitor 0.1.5-1+deb10u1 to buster-wikimedia (T213527) [production]
15:31 <_joe_> upgraded all php extensions to php 7.2 compatible versions on mwmaint1002 [production]