9001-9050 of 10000 results (59ms)
2016-10-10 §
14:03 <gehel> reimage maps-test200[34] - T147194 [production]
13:38 <marostegui> Dropping hitcounter, _counter memory tables in S4 - db1040 (master) - T132837 [production]
13:20 <zeljkof> ending EU SWAT! [production]
13:15 <zfilipin@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:314257|Activate subphrases autocomplete on wikisources, mw.org and wikitech (T146208)]] (duration: 00m 50s) [production]
13:12 <gehel> reimage maps-test2002 - T147194 [production]
12:47 <moritzm> uploaded nodejs 4.6.0 for jessie-wikimedia to carbon [production]
12:44 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Restore original weight for db1082 after its RAID controller firmware - T145533 (duration: 00m 55s) [production]
12:06 <hoo@tin> Synchronized php-1.28.0-wmf.21/extensions/Wikidata: Update Wikibase, add EntityHandler::supportsCategories (T147748) (duration: 02m 25s) [production]
11:52 <godog> swift eqiad-prod: ms-be1022 to weight 2000 T136631 [production]
11:33 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase weight for db1082 after its RAID controller firmware - T145533 (duration: 00m 49s) [production]
10:52 <marostegui> Dropping hitcounter, _counter memory tables in S2 - db1069 - T132837 [production]
10:42 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1082 with some small weight after its RAID controller firmware - T145533 (duration: 00m 50s) [production]
10:04 <jynus> running ALTER TABLE search_documentfield ENGINE=InnoDB, FORCE; on phabricato db replica (db1043) [production]
09:39 <marostegui> Dropping hitcounter, _counter memory tables in S2 - db1063- T132837 [production]
09:32 <moritzm> rolling reboot of swift frontend servers in eqiad for kernel security update [production]
09:30 <moritzm> pruning older, unused kernel images on labstore1003 [production]
08:57 <marostegui> Dropping hitcounter, _counter memory tables in S2 - dbstore1002 - T132837 [production]
08:56 <jynus> reboot db1043 to test new mysql configuration and general upgrade- proxy will complain [production]
08:53 <marostegui> Dropping hitcounter, _counter memory tables in S2 - dbstore1001 - T132837 [production]
08:53 <godog> reboot graphite2001 and graphite1001 for trusty kernel upgrade [production]
08:40 <hoo> Populated the sites/ site_identifiers tables on olowiki (T146614) [production]
08:38 <marostegui> Dropping hitcounter, _counter memory tables in S2 - dbstore2002 - T132837 [production]
08:12 <akosiaris> clear mx1001's queues from backscatter spam T147173 [production]
07:34 <marostegui> Deploying schema change on S4 codfw only commonswiki.revision - T147305 [production]
07:29 <moritzm> installing php security updates on jessie systems [production]
06:15 <marostegui> db1082: Upgrading RAID controller firmware [production]
06:13 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1082 to upgrade its RAID controller firmware - T145533 (duration: 00m 50s) [production]
05:42 <jynus> reseting slave on es2 eqiad master (es1015) [production]
05:11 <jynus@tin> Synchronized wmf-config/db-codfw.php: mariadb: Depool es2015 (master, crashed); replaced by es2016 (duration: 00m 49s) [production]
04:50 <jynus> changing topology of es2 @ codfw [production]
02:30 <l10nupdate@tin> ResourceLoader cache refresh completed at Mon Oct 10 02:30:47 UTC 2016 (duration 4m 51s) [production]
02:25 <mwdeploy@tin> scap sync-l10n completed (1.28.0-wmf.21) (duration: 09m 40s) [production]
2016-10-09 §
09:07 <elukey> chmod o+r /var/lib/varnish/frontend/_.vsm and /var/lib/varnish/cp2008/_.vsm on cp2008 to avoid gmond errors [production]
09:01 <jynus> dropping unneded files on db1026 to mitigate disk issues for the next week [production]
08:45 <elukey> powercycling cp2008, no ssh and mgmt console frozen [production]
02:27 <l10nupdate@tin> ResourceLoader cache refresh completed at Sun Oct 9 02:27:12 UTC 2016 (duration 4m 36s) [production]
02:22 <mwdeploy@tin> scap sync-l10n completed (1.28.0-wmf.21) (duration: 08m 59s) [production]
2016-10-08 §
19:48 <bd808> cp2008 Strongswan failures for both ipv4 and ipv6 across a larg number (all?) hosts [production]
09:40 <elukey> masked the kafka systemd unit on kafka1018 and re-enabling puppet [production]
09:10 <apergos> puppet disabled on kafka1018, leave broker down, bad disk /dev/sdi (see dmesg for sample errors) [production]
02:31 <l10nupdate@tin> ResourceLoader cache refresh completed at Sat Oct 8 02:31:15 UTC 2016 (duration 6m 20s) [production]
02:24 <mwdeploy@tin> scap sync-l10n completed (1.28.0-wmf.21) (duration: 09m 09s) [production]
2016-10-07 §
23:16 <Krenair> test morebots [production]
23:15 <thcipriani@tin> Synchronized w/robots.php: Revert change to robots.php (duration: 00m 49s) [production]
22:22 <mutante> etcd servers have puppet issue with Etcd_user[root] [production]
22:04 <ejegg> updated payments-wiki from 1fbe171bca30d1fb7a4b7d937a740d87f7c9c8e3 to b4ad60e739b9dbb97f08a3623db961a74682422a [production]
20:30 <bblack> lead.wikimedia.org: replaced by cobalt functionally, please leave it untouched for now with puppet disabled! [production]
20:23 <twentyafterfour> restarted phd on iridium [production]
20:08 <twentyafterfour> disabled phd + puppet on iridium and scheduled downtime in icinga to silence alerts [production]
19:46 <mutante> deleted old /var/lib/gerrit2/ data on cobalt, syncing from lead [production]