9151-9200 of 10000 results (53ms)
2016-10-10 §
05:11 <jynus@tin> Synchronized wmf-config/db-codfw.php: mariadb: Depool es2015 (master, crashed); replaced by es2016 (duration: 00m 49s) [production]
04:50 <jynus> changing topology of es2 @ codfw [production]
02:30 <l10nupdate@tin> ResourceLoader cache refresh completed at Mon Oct 10 02:30:47 UTC 2016 (duration 4m 51s) [production]
02:25 <mwdeploy@tin> scap sync-l10n completed (1.28.0-wmf.21) (duration: 09m 40s) [production]
2016-10-09 §
09:07 <elukey> chmod o+r /var/lib/varnish/frontend/_.vsm and /var/lib/varnish/cp2008/_.vsm on cp2008 to avoid gmond errors [production]
09:01 <jynus> dropping unneded files on db1026 to mitigate disk issues for the next week [production]
08:45 <elukey> powercycling cp2008, no ssh and mgmt console frozen [production]
02:27 <l10nupdate@tin> ResourceLoader cache refresh completed at Sun Oct 9 02:27:12 UTC 2016 (duration 4m 36s) [production]
02:22 <mwdeploy@tin> scap sync-l10n completed (1.28.0-wmf.21) (duration: 08m 59s) [production]
2016-10-08 §
19:48 <bd808> cp2008 Strongswan failures for both ipv4 and ipv6 across a larg number (all?) hosts [production]
09:40 <elukey> masked the kafka systemd unit on kafka1018 and re-enabling puppet [production]
09:10 <apergos> puppet disabled on kafka1018, leave broker down, bad disk /dev/sdi (see dmesg for sample errors) [production]
02:31 <l10nupdate@tin> ResourceLoader cache refresh completed at Sat Oct 8 02:31:15 UTC 2016 (duration 6m 20s) [production]
02:24 <mwdeploy@tin> scap sync-l10n completed (1.28.0-wmf.21) (duration: 09m 09s) [production]
2016-10-07 §
23:16 <Krenair> test morebots [production]
23:15 <thcipriani@tin> Synchronized w/robots.php: Revert change to robots.php (duration: 00m 49s) [production]
22:22 <mutante> etcd servers have puppet issue with Etcd_user[root] [production]
22:04 <ejegg> updated payments-wiki from 1fbe171bca30d1fb7a4b7d937a740d87f7c9c8e3 to b4ad60e739b9dbb97f08a3623db961a74682422a [production]
20:30 <bblack> lead.wikimedia.org: replaced by cobalt functionally, please leave it untouched for now with puppet disabled! [production]
20:23 <twentyafterfour> restarted phd on iridium [production]
20:08 <twentyafterfour> disabled phd + puppet on iridium and scheduled downtime in icinga to silence alerts [production]
19:46 <mutante> deleted old /var/lib/gerrit2/ data on cobalt, syncing from lead [production]
19:45 <mutante> rsyncing /var/lib/gerrit2 from lead to cobalt [production]
19:30 <mutante> removed gerrit IPs from cobalt interfaces [production]
19:29 <mutante> disabled puppet on lead and cobalt [production]
19:21 <mutante> re-enabling puppet on cobalt [production]
19:21 <mutante> removed gerrit IP from lead's interface, v4 and v6 [production]
19:09 <mutante> rsyncing gerrit data one more time from lead to cobalt [production]
19:08 <cmjohnson1> db1065 swapping failed disk slot 9 T147396 [production]
19:07 <ostriches> stopped puppet on lead [production]
19:07 <mutante> stopping gerrit on lead [production]
19:02 <mutante> cobalt, disabled puppet, removed service IP from interface [production]
17:28 <mutante> rsyncing gerrit data from lead to cobalt [production]
16:53 <jynus> testing img_metadata nuking for T145953 and T147015 (backups on neodymium) [production]
16:25 <gehel> reimage maps-test2001 - T147194 [production]
16:14 <akosiaris> build python-irclib for jessie and upload it to apt.wikimedia.org jessie-wikimedia/main [production]
16:06 <moritzm> updated hhvm package for jessie to 3.12.9 [production]
14:41 <moritzm> uploaded openssl 1.0.2j for jessie-wikimedia to carbon [production]
14:41 <kart_> Update cxserver to fa2f715 (T147552) [production]
13:56 <elukey> reimaing mw123[45] to Debian Jessie (last two api appservers) [production]
12:39 <elukey> reimaging mw123[23] to Debian Jessie [production]
12:09 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1082 with its original weight - T145533 (duration: 00m 52s) [production]
11:17 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1082 with a bit less weight than usual to start with - T145533 (duration: 00m 55s) [production]
10:52 <moritzm> reimaging mw1238, mw1239 to jessie [production]
10:28 <elukey> reimaging mw123[01] to Debian Jessie [production]
10:27 <elukey> mw122[89] back in live api server pool [production]
10:00 <_joe_> updated conftool to 0.3.1 on all the cluster except caches, T147480 [production]
09:48 <_joe_> creating etcd100[1-6].eqiad.wmnet on ganeti, T147620 [production]
09:32 <moritzm> reimaging mw1220, mw1236, mw1237 to jessie [production]
09:28 <moritzm> installing pillow/python-imaging security updates on Ubuntu systems [production]