601-650 of 10000 results (36ms)
2016-10-10 §
11:52 <godog> swift eqiad-prod: ms-be1022 to weight 2000 T136631 [production]
11:33 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase weight for db1082 after its RAID controller firmware - T145533 (duration: 00m 49s) [production]
10:52 <marostegui> Dropping hitcounter, _counter memory tables in S2 - db1069 - T132837 [production]
10:42 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1082 with some small weight after its RAID controller firmware - T145533 (duration: 00m 50s) [production]
10:04 <jynus> running ALTER TABLE search_documentfield ENGINE=InnoDB, FORCE; on phabricato db replica (db1043) [production]
09:39 <marostegui> Dropping hitcounter, _counter memory tables in S2 - db1063- T132837 [production]
09:32 <moritzm> rolling reboot of swift frontend servers in eqiad for kernel security update [production]
09:30 <moritzm> pruning older, unused kernel images on labstore1003 [production]
08:57 <marostegui> Dropping hitcounter, _counter memory tables in S2 - dbstore1002 - T132837 [production]
08:56 <jynus> reboot db1043 to test new mysql configuration and general upgrade- proxy will complain [production]
08:53 <marostegui> Dropping hitcounter, _counter memory tables in S2 - dbstore1001 - T132837 [production]
08:53 <godog> reboot graphite2001 and graphite1001 for trusty kernel upgrade [production]
08:40 <hoo> Populated the sites/ site_identifiers tables on olowiki (T146614) [production]
08:38 <marostegui> Dropping hitcounter, _counter memory tables in S2 - dbstore2002 - T132837 [production]
08:12 <akosiaris> clear mx1001's queues from backscatter spam T147173 [production]
07:34 <marostegui> Deploying schema change on S4 codfw only commonswiki.revision - T147305 [production]
07:29 <moritzm> installing php security updates on jessie systems [production]
06:15 <marostegui> db1082: Upgrading RAID controller firmware [production]
06:13 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1082 to upgrade its RAID controller firmware - T145533 (duration: 00m 50s) [production]
05:42 <jynus> reseting slave on es2 eqiad master (es1015) [production]
05:11 <jynus@tin> Synchronized wmf-config/db-codfw.php: mariadb: Depool es2015 (master, crashed); replaced by es2016 (duration: 00m 49s) [production]
04:50 <jynus> changing topology of es2 @ codfw [production]
02:30 <l10nupdate@tin> ResourceLoader cache refresh completed at Mon Oct 10 02:30:47 UTC 2016 (duration 4m 51s) [production]
02:25 <mwdeploy@tin> scap sync-l10n completed (1.28.0-wmf.21) (duration: 09m 40s) [production]
2016-10-09 §
09:07 <elukey> chmod o+r /var/lib/varnish/frontend/_.vsm and /var/lib/varnish/cp2008/_.vsm on cp2008 to avoid gmond errors [production]
09:01 <jynus> dropping unneded files on db1026 to mitigate disk issues for the next week [production]
08:45 <elukey> powercycling cp2008, no ssh and mgmt console frozen [production]
02:27 <l10nupdate@tin> ResourceLoader cache refresh completed at Sun Oct 9 02:27:12 UTC 2016 (duration 4m 36s) [production]
02:22 <mwdeploy@tin> scap sync-l10n completed (1.28.0-wmf.21) (duration: 08m 59s) [production]
2016-10-08 §
19:48 <bd808> cp2008 Strongswan failures for both ipv4 and ipv6 across a larg number (all?) hosts [production]
09:40 <elukey> masked the kafka systemd unit on kafka1018 and re-enabling puppet [production]
09:10 <apergos> puppet disabled on kafka1018, leave broker down, bad disk /dev/sdi (see dmesg for sample errors) [production]
02:31 <l10nupdate@tin> ResourceLoader cache refresh completed at Sat Oct 8 02:31:15 UTC 2016 (duration 6m 20s) [production]
02:24 <mwdeploy@tin> scap sync-l10n completed (1.28.0-wmf.21) (duration: 09m 09s) [production]
2016-10-07 §
23:16 <Krenair> test morebots [production]
23:15 <thcipriani@tin> Synchronized w/robots.php: Revert change to robots.php (duration: 00m 49s) [production]
22:22 <mutante> etcd servers have puppet issue with Etcd_user[root] [production]
22:04 <ejegg> updated payments-wiki from 1fbe171bca30d1fb7a4b7d937a740d87f7c9c8e3 to b4ad60e739b9dbb97f08a3623db961a74682422a [production]
20:30 <bblack> lead.wikimedia.org: replaced by cobalt functionally, please leave it untouched for now with puppet disabled! [production]
20:23 <twentyafterfour> restarted phd on iridium [production]
20:08 <twentyafterfour> disabled phd + puppet on iridium and scheduled downtime in icinga to silence alerts [production]
19:46 <mutante> deleted old /var/lib/gerrit2/ data on cobalt, syncing from lead [production]
19:45 <mutante> rsyncing /var/lib/gerrit2 from lead to cobalt [production]
19:30 <mutante> removed gerrit IPs from cobalt interfaces [production]
19:29 <mutante> disabled puppet on lead and cobalt [production]
19:21 <mutante> re-enabling puppet on cobalt [production]
19:21 <mutante> removed gerrit IP from lead's interface, v4 and v6 [production]
19:09 <mutante> rsyncing gerrit data one more time from lead to cobalt [production]
19:08 <cmjohnson1> db1065 swapping failed disk slot 9 T147396 [production]
19:07 <ostriches> stopped puppet on lead [production]