3951-4000 of 10000 results (67ms)
2019-09-23 §
07:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1123 to change binlog format T230783', diff saved to https://phabricator.wikimedia.org/P9145 and previous config saved to /var/cache/conftool/dbconfig/20190923-070628-marostegui.json [production]
06:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Change db1123 and db1078 roles, db1078 will serve logpager and recentchanges, db1123 will just serve general traffic', diff saved to https://phabricator.wikimedia.org/P9144 and previous config saved to /var/cache/conftool/dbconfig/20190923-065056-marostegui.json [production]
05:23 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db1066 from config T233071 (duration: 00m 56s) [production]
05:22 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db1066 from config T233071 (duration: 01m 15s) [production]
2019-09-22 §
19:07 <volans|off> marostegui set s3 master RW [production]
2019-09-21 §
05:42 <shdubsh> re-enable input-kafka-rsyslog-shipper in codfw [production]
05:33 <shdubsh> drop input-kafka-rsyslog-shipper in codfw [production]
02:15 <bblack> dbproxy1017: executing "systemctl reload haproxy" to recover from false healthcheck failure (network issues) on master [production]
02:14 <bblack> dbproxy1016: executing "systemctl reload haproxy" to recover from false healthcheck failure (network issues) on master [production]
01:52 <shdubsh> temporarily removing input-kafka-rsyslog-shipper-eqiad/codfw from logstash2004-5-6 [production]
01:34 <mutante> restarting mobileapps service on scb* [production]
01:34 <mutante> restarted mobileapps service on scb1001 [production]
01:21 <bblack@cumin1001> conftool action : set/pooled=yes; selector: name=cp1087.eqiad.wmnet [production]
01:21 <bblack@cumin1001> conftool action : set/pooled=yes; selector: name=cp1088.eqiad.wmnet [production]
01:21 <bblack> re-pooling cp108[78] in D2 via confctl [production]
01:14 <shdubsh> temporarily removing input-kafka-rsyslog-shipper-eqiad/codfw from logstash1007 [production]
01:08 <shdubsh> removed input-kafka-rsyslog-shipper-eqiad/codfw from logstash inputs logstash1008 and logstash1009 [production]
00:54 <mutante> aqs1009 - systemctl restart aqs [production]
00:54 <mutante> aqs1006 - systemctl restart aqs [production]
00:48 <mutante> aqs1005 - systemctl restart aqs [production]
00:46 <shdubsh> restarting logstash on logstash1008 without udp-localhost-eqiad/codfw configs [production]
00:39 <bblack@cumin1001> conftool action : set/pooled=no; selector: name=cp1088.eqiad.wmnet [production]
00:38 <bblack> depooling confctl things in rack D2 [production]
00:38 <bblack@cumin1001> conftool action : set/pooled=no; selector: name=cp1087.eqiad.wmnet [production]
2019-09-20 §
21:30 <urbanecm@deploy1001> Synchronized php-1.34.0-wmf.22/extensions/CheckUser: fix T233453 (duration: 00m 56s) [production]
21:29 <urbanecm@deploy1001> Synchronized php-1.34.0-wmf.23/extensions/CheckUser: fix T233453 (duration: 00m 58s) [production]
19:26 <XioNoX> update eqsin firewall filters - T233268 [production]
16:35 <krinkle@deploy1001> Synchronized vendor/: ead70240892e9 (duration: 00m 59s) [production]
16:14 <XioNoX> update eqiad firewall filters - T233268 [production]
16:11 <XioNoX> update esams firewall filters - T233268 [production]
15:17 <Urbanecm> mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=bgwiki --logwiki=metawiki 'Newrdkter' 'NRdk' (T233313) [production]
15:03 <XioNoX> remove AS-PATH prepending in ams [production]
11:29 <@> helmfile [STAGING] Ran 'sync' command on namespace 'restrouter' for release 'staging' . [production]
11:22 <@> helmfile [STAGING] Ran 'sync' command on namespace 'restrouter' for release 'staging' . [production]
11:22 <@> helmfile [STAGING] Ran 'sync' command on namespace 'restrouter' for release 'staging' . [production]
11:22 <@> helmfile [STAGING] Ran 'sync' command on namespace 'restrouter' for release 'staging' . [production]
11:16 <@> helmfile [STAGING] Ran 'sync' command on namespace 'restrouter' for release 'staging' . [production]
11:15 <@> helmfile [STAGING] Ran 'sync' command on namespace 'restrouter' for release 'staging' . [production]
10:17 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) [production]
10:17 <jbond@cumin1001> Updating IPMI password on 1 hosts - jbond@cumin1001 [production]
10:17 <jbond@cumin1001> START - Cookbook sre.hosts.ipmi-password-reset [production]
10:17 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) [production]
10:17 <jbond@cumin1001> Updating IPMI password on 1 hosts - jbond@cumin1001 [production]
10:17 <jbond@cumin1001> START - Cookbook sre.hosts.ipmi-password-reset [production]
10:17 <jbond@cumin1001> END (FAIL) - Cookbook sre.hosts.ipmi-password-reset (exit_code=99) [production]
10:17 <jbond@cumin1001> START - Cookbook sre.hosts.ipmi-password-reset [production]
09:31 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) [production]
09:31 <jbond@cumin1001> Updating IPMI password on 1 hosts - jbond@cumin1001 [production]
09:31 <jbond@cumin1001> START - Cookbook sre.hosts.ipmi-password-reset [production]
09:30 <jbond@cumin1001> END (ERROR) - Cookbook sre.hosts.ipmi-password-reset (exit_code=97) [production]