851-900 of 10000 results (54ms)
2019-02-15 §
07:55 <godog> bounce prometheus@ops on prometheus2004 to take a snapshot [production]
06:40 <marostegui> Stop puppet on labsdb1005 to leave "max_user_connections" on my.cnf - T216170 T216208 [production]
06:39 <marostegui> Restart labsdb1005 with max_user_connections = 20 T216208 [production]
06:17 <marostegui> Deploy schema change on db1109 - T210713 [production]
06:16 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1109 (duration: 00m 49s) [production]
06:13 <marostegui> Reload haproxy on dbproxy11 to repool labsdb1009 [production]
00:39 <mutante> puppetmaster1001: sudo puppet node clean bast3003.wikimedia.org ; sudo puppet node deactivate bast3003.wikimedia.org (T216199) [production]
00:15 <jynus> setting labsdb1005 back into read-write [production]
2019-02-14 §
23:47 <jynus> restarting labsdb1005 mysql in read only mode [production]
23:37 <niharika29@deploy1001> Finished deploy [scholarships/scholarships@25ea138]: Update app with updated dependencies to mitigate PHPMailer error T215302 (duration: 00m 02s) [production]
23:37 <niharika29@deploy1001> Started deploy [scholarships/scholarships@25ea138]: Update app with updated dependencies to mitigate PHPMailer error T215302 [production]
22:07 <andrewbogott> rebuilding labvirt1012 as cloudvirt1012, T216190 [production]
20:38 <bstorm_> Restarted mariadb on labsdb1005 for https://wikitech.wikimedia.org/wiki/Incident_documentation/20190214-labsdb1005 [production]
20:18 <thcipriani> thcipriani@deploy1001 rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.17 [production]
20:14 <thcipriani@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.17 [production]
20:09 <ejegg> updated fundraising CiviCRM from 02ea871b88 to 165fbf5894 [production]
19:42 <thcipriani@deploy1001> Synchronized php-1.33.0-wmf.17/extensions/GrowthExperiments/modules/help: SWAT: [[gerrit:490674|Help Panel: Fix IME broken in help panel search]] T216131 (duration: 00m 54s) [production]
19:14 <thcipriani@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:487007|Stop NavPopups gadget conflict with PagePreviews on Wikivoyage]] T214878 (duration: 00m 54s) [production]
19:01 <mutante> scandium - deleting parsoid clone dir and running puppet one more time, to fix permissions to allow wikidev [production]
18:52 <mutante> scandium - deleting parsoid clone dir and running puppet one more time, to fix permissions to allow wikidev [production]
18:12 <mutante> scandium - deleting parsoid clone dir and running puppet [production]
18:03 <fsero> upgrading tiller to 2.12.2 on eqiad [production]
17:34 <godog> bounce rsyslog on wezen/lithium, tls listener timeout in icinga [production]
16:59 <moritzm> restarting apertium-apy on scb1001 to pick up Python security update [production]
16:39 <marostegui> Depool labsdb1009 - T210713 [production]
16:26 <fsero> upgrading tiller on codfw [production]
16:11 <fsero> updating tiller version on staging cluster [production]
16:10 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Repool db2085 - T214840 (duration: 00m 52s) [production]
15:50 <fsero> building and publishing new tiller docker image on boron [production]
15:50 <END> (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) (volans@cumin1001) [production]
15:43 <START> - Cookbook sre.hosts.upgrade-and-reboot (volans@cumin1001) [production]
15:28 <volans> upgraded spicerack to v0.0.15 on cumin[12]001 [production]
15:26 <volans> uploaded spicerack_0.0.15-1_amd64.deb to apt.wikimedia.org stretch-wikimedia [production]
15:12 <marostegui> Clear idrac logs from db2085 - T214840 [production]
14:45 <godog> depool and stop logstash1009 for stretch reimage - T213898 [production]
14:20 <marostegui> Stop MySQL on db2085 for on-site maintenance - T214840 [production]
14:12 <jijiki> Enabling puppet on thumbor* servers - T214597 [production]
13:39 <arturo> T215892 icinga downtime cloudvirt1024 for 2 weeks [production]
12:22 <zeljkof> EU SWAT finished [production]
12:21 <zfilipin@deploy1001> Synchronized php-1.33.0-wmf.17/extensions/ExternalGuidance/: SWAT: [[gerrit:490523|Fix the eventlogging schema definition as per manifest_version=2]] (duration: 00m 55s) [production]
11:43 <_joe_> restarting hhvm on mw1338, hot tc exhausted T216084 [production]
11:04 <_joe_> upgrading python3-etcd on stretch T209136 [production]
11:03 <jbond42> rolling security updates for curl [production]
11:02 <jijiki> Disabling puppet on thumbor* servers - T214597 [production]
10:59 <moritzm> installing python3.4 security updates [production]
10:53 <godog> bounce prometheus instances on prometheus2004 to take a snapshot [production]
08:10 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1106 T214840 (duration: 00m 52s) [production]
07:57 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1087 T210713 (duration: 00m 54s) [production]
07:36 <marostegui> Stop MySQL on db1106 for reboot - T214840 [production]
06:10 <marostegui> Deploy schema change on db1087 with replication, lag will be generated on labsdb:s8 T210713 [production]