7501-7550 of 10000 results (83ms)
2019-08-20 §
10:51 <marostegui> Stop MySQL on db2051 and db2056 for decommission T230777 T230778 [production]
10:30 <ema> cp5002: restart trafficserver for compress.so config change [production]
10:11 <tarrow> termbox 2nd smoketests finished [production]
09:52 <marostegui> Remove db2051 and db2056 from tendril and zarcillo - T230777 T230778 [production]
09:35 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db2051 and db2056 from config T230777 T230778 (duration: 00m 48s) [production]
09:00 <tarrow> Starting 2nd smoketest of termbox service on eqiad: T229907 [production]
08:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Set db2123 as codfw s5 master - T230106', diff saved to https://phabricator.wikimedia.org/P8936 and previous config saved to /var/cache/conftool/dbconfig/20190820-082802-marostegui.json [production]
08:24 <marostegui@cumin1001> dbctl commit (dc=all): 'Re-organize s5 codfw weights - T230106', diff saved to https://phabricator.wikimedia.org/P8935 and previous config saved to /var/cache/conftool/dbconfig/20190820-082411-marostegui.json [production]
08:19 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Promote db2123 to s5 codfw master T230106 (duration: 00m 48s) [production]
08:05 <marostegui> Switchover s5 codfw master db2052 -> db2123 T230106 [production]
07:50 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Reorganize s5 codfw weights T230106 (duration: 00m 47s) [production]
07:49 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2059 and db2066, those two will be decommissioned T228258', diff saved to https://phabricator.wikimedia.org/P8934 and previous config saved to /var/cache/conftool/dbconfig/20190820-074900-marostegui.json [production]
06:59 <moritzm> installing failoid1001/2001 T229903 [production]
05:59 <marostegui> Stop MySQL and shutdown db1114 for on-siste maintenance - T229452 [production]
05:55 <marostegui> Stop MySQL on db2044 for decommissioning - T221594 [production]
05:36 <marostegui> Remove db2049 from tendril and zarcillo T230721 [production]
05:35 <marostegui> Stop MySQL on db2049 for decommissioning - T230721 [production]
05:24 <marostegui> Reload haproxy on dbproxy2002 T230705 [production]
05:18 <marostegui> Switchover m2 codfw master, db2044 -> db2067 T230705 [production]
2019-08-19 §
21:21 <ejegg> updated payments-wiki from 7b8091ba87 to 85dce8f79f [production]
21:21 <ejegg> updated payments-wiki subdesarrollo [production]
19:35 <ejegg> updated payments-wiki from e3b378f65d to 7b8091ba87 [production]
18:57 <Urbanecm> Morning SWaT done [production]
18:48 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: Raise rollback limit for all groups (T228708) (duration: 00m 48s) [production]
18:36 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 26317c7: Fix zhwikisource wgExtraNamespaces entry (T230294) (duration: 00m 48s) [production]
18:31 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: b21bbc0: Add `WS` and `CAT` as aliases for zhwikisource namespaces (T230548) (duration: 00m 47s) [production]
18:26 <urbanecm@deploy1001> Synchronized wmf-config/CommonSettings.php: SWAT: 0a87e3c: Assign all rights assigned to suppress group to oversight group (T230601) (duration: 00m 48s) [production]
17:56 <ebernhar1son> freeze cloudelastic writes to let prod clear 30 min backlog [production]
17:23 <onimisionipe@deploy1001> Finished deploy [wdqs/wdqs@2d36896]: Fix Blazegraph dictionary mixup (duration: 18m 18s) [production]
17:17 <shdubsh> restarting icinga to disable UI autocomplete [production]
17:04 <onimisionipe@deploy1001> Started deploy [wdqs/wdqs@2d36896]: Fix Blazegraph dictionary mixup [production]
16:45 <onimisionipe> pool elastic2050. mgmt issue has been resolved - T230597 [production]
15:39 <ejegg> updated payments-wiki from 00eb090dcc to e3b378f65d [production]
13:57 <vgutierrez> repooling cp5001 [production]
12:51 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db2049 from config T230721 (duration: 00m 48s) [production]
12:50 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db2049 from config T230721 (duration: 00m 48s) [production]
12:38 <vgutierrez> depooling cp5001 prior to ats-tls deployment [production]
12:02 <Urbanecm> EU SWAT done [production]
11:46 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Revert 483691c (T225053) (duration: 00m 48s) [production]
11:39 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 483691c: Revert "Revert "Switch property terms migration to WRITE_NEW on client wikis"" (T225053) (duration: 00m 48s) [production]
11:15 <jmm@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
11:12 <jmm@cumin1001> START - Cookbook sre.ganeti.makevm [production]
11:03 <jmm@cumin2001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
11:02 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
11:00 <jmm@cumin2001> START - Cookbook sre.ganeti.makevm [production]
10:53 <elukey@cumin1001> START - Cookbook sre.ganeti.makevm [production]
10:53 <elukey@cumin1001> END (ERROR) - Cookbook sre.ganeti.makevm (exit_code=97) [production]
10:52 <elukey@cumin1001> START - Cookbook sre.ganeti.makevm [production]
10:38 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:530826| Bumping portals to master (T128546)]] (duration: 00m 49s) [production]
10:37 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:530826| Bumping portals to master (T128546)]] (duration: 00m 49s) [production]