|
2018-10-10
§
|
| 14:05 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) (volans@neodymium) |
[production] |
| 14:01 |
<START> |
- Cookbook sre.switchdc.mediawiki.00-warmup-caches (volans@neodymium) |
[production] |
| 14:01 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) (volans@neodymium) |
[production] |
| 14:01 |
<START> |
- Cookbook sre.switchdc.mediawiki.00-reduce-ttl (volans@neodymium) |
[production] |
| 14:00 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) (volans@neodymium) |
[production] |
| 14:00 |
<START> |
- Cookbook sre.switchdc.mediawiki.00-disable-puppet (volans@neodymium) |
[production] |
| 12:18 |
<_joe_> |
decommissioning conf1001-1003: stopping etcd, nginx, and masking both |
[production] |
| 11:41 |
<jynus> |
renaming some s3 wiki tables on eqiad master to prevent split brain T184805 |
[production] |
| 11:29 |
<zeljkof> |
EU SWAT finished |
[production] |
| 11:26 |
<zfilipin@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:465602|Permissions changes on itwikibooks (T206447)]] (duration: 00m 57s) |
[production] |
| 10:54 |
<marostegui> |
Set a replication filter on db1075 (s3 eqiad) to ignore enwikivoyage, cebwiki, shwiki, srwiki & mgwiktionary - T184805 |
[production] |
| 10:49 |
<marostegui@deploy1001> |
Synchronized dblists/s5.dblist: Update s5.dblist to reflect the wikis moved from s3 - T184805 (duration: 00m 56s) |
[production] |
| 10:48 |
<marostegui@deploy1001> |
Synchronized dblists/s3.dblist: Update s3.dblist to reflect the wikis moved to s5 - T184805 (duration: 00m 58s) |
[production] |
| 09:12 |
<ema> |
Traffic: move restbase back to eqiad T203777 |
[production] |
| 09:07 |
<ema> |
Traffic: set services active/active T203777 |
[production] |
| 09:00 |
<ema> |
Traffic: route esams caches back to eqiad T203777 |
[production] |
| 08:27 |
<moritzm> |
installing fuse security updates |
[production] |
| 08:07 |
<ariel@deploy1001> |
Finished deploy [dumps/dumps@0714a93]: fix adds/changes dumps generation when prev run is missing (duration: 00m 06s) |
[production] |
| 08:07 |
<ariel@deploy1001> |
Started deploy [dumps/dumps@0714a93]: fix adds/changes dumps generation when prev run is missing |
[production] |
| 08:01 |
<moritzm> |
rolling out debdeploy 0.0.99.6 |
[production] |
| 07:51 |
<elukey> |
cleaned up some log files from eventlog1002 |
[production] |
| 02:55 |
<ejegg> |
updated payments-wiki from 1472604b6e to 7fb1aae963 |
[production] |
| 00:19 |
<krinkle@deploy1001> |
Synchronized php-1.32.0-wmf.24/includes/utils/UIDGenerator.php: T94522 - I2a0c51bea58 (duration: 00m 56s) |
[production] |
| 00:15 |
<krinkle@deploy1001> |
sync-file aborted: T205567 - I75f1eb6dc2cb (duration: 00m 01s) |
[production] |
| 00:14 |
<krinkle@deploy1001> |
Synchronized php-1.32.0-wmf.24/tests/phpunit/includes/utils/: T94522 - I2a0c51bea58 (duration: 01m 02s) |
[production] |
|
2018-10-09
§
|
| 22:58 |
<SMalyshev> |
repooled wdqs2003 |
[production] |
| 22:26 |
<shdubsh> |
repairing /dev/sdl1 on ms-be2040 - T199198 |
[production] |
| 21:52 |
<bblack> |
cp1085: varnish backend restart for mbox lag |
[production] |
| 21:50 |
<mutante> |
releases1001 - restarted jenkins (it went from 200 -> 503 -> 403) curl localhost:8080 works again after restart, icinga check still getting 403 now |
[production] |
| 21:41 |
<ejegg|food> |
updated fundraising CiviCRM from 7a0d14015e to 1165e7ed79 |
[production] |
| 20:08 |
<mutante> |
repair /dev/sdg1 on ms-be2041 - T199198 |
[production] |
| 19:37 |
<XioNoX> |
disable igmp-snooping on asw2-c-eqiad - T201039 |
[production] |
| 19:25 |
<XioNoX> |
disable igmp-snooping on asw2-b-eqiad - T201039 |
[production] |
| 19:20 |
<XioNoX> |
bounce igmp-snooping on asw2-b-eqiad |
[production] |
| 18:24 |
<ottomata> |
adding Accept header to all varnishkafka generated webrequest logs |
[production] |
| 17:21 |
<SMalyshev> |
depooled wdq23 again, sigh |
[production] |
| 13:54 |
<moritzm> |
rebooting prometheus1004 for kernel security update |
[production] |
| 13:41 |
<moritzm> |
rebooting prometheus1003 for kernel security update |
[production] |
| 13:28 |
<moritzm> |
rebooting prometheus2004 for kernel security update |
[production] |
| 13:13 |
<moritzm> |
rebooting prometheus2003 for kernel security update |
[production] |
| 12:54 |
<gehel> |
silencing wdqs-public lag alerts (service still functional, and SLO unclear) - T199228 |
[production] |
| 12:45 |
<moritzm> |
installing imagemagick security updates |
[production] |
| 11:47 |
<END> |
(ERROR) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=2) (volans@neodymium) |
[production] |
| 11:46 |
<START> |
- Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (volans@neodymium) |
[production] |
| 11:45 |
<akosiaris> |
dry-run services switchover from codfw to eqiad in preparation for Thursday |
[production] |
| 11:37 |
<END> |
(ERROR) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=2) (volans@neodymium) |
[production] |
| 11:37 |
<START> |
- Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (volans@neodymium) |
[production] |
| 11:14 |
<volans> |
live-test of the inverted switchdc (eqiad->codfw) completed, all good - T203777 |
[production] |
| 11:14 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.08-update-tendril (exit_code=0) (volans@neodymium) |
[production] |
| 11:13 |
<START> |
- Cookbook sre.switchdc.mediawiki.08-update-tendril (volans@neodymium) |
[production] |