|
2019-12-04
§
|
| 09:29 |
<godog> |
roll-restart logstash7 in codfw/eqiad after https://gerrit.wikimedia.org/r/c/operations/puppet/+/554472 |
[production] |
| 09:15 |
<marostegui> |
Reload labsdb1010 after reimporting wikidatawiki.page - T238399 |
[production] |
| 09:06 |
<moritzm> |
updated jenkins on apt.wikimedia.org to 2.190.3 (T239586) |
[production] |
| 08:05 |
<effie> |
Restart php7-fpm on mw1348 |
[production] |
| 07:09 |
<marostegui> |
Depool labsdb1010 to reimport wikidatawiki.page - T238399 |
[production] |
| 07:02 |
<marostegui> |
Repool labsdb1011 |
[production] |
| 06:36 |
<mutante> |
removed LVS IP for git-ssh from interface on phab1003 |
[production] |
| 06:25 |
<dzahn@cumin1001> |
conftool action : set/weight=10; selector: name=phab1001-vcs.eqiad.wmnet |
[production] |
| 06:13 |
<mutante> |
phab1001 - running rsync of /srv/repos with --delete because it's larger than the source by about 5GB - deleting objects to match phab1003, former prod server. now both 50G (T238956) |
[production] |
| 06:04 |
<marostegui> |
Depool labsdb1011 |
[production] |
| 06:01 |
<mutante> |
rsyncing /srv/repos data once again. pulling from phab1003 to phab1001 (T238956) |
[production] |
| 05:51 |
<marostegui> |
Deploy schema change on s3 primary master (db1123) |
[production] |
| 04:59 |
<mutante> |
removed downtime for phabricator.wikimedia.org meta service (paging) |
[production] |
| 04:58 |
<mutante> |
phabricator maintenance ended for today - now running on phab1001 (buster) |
[production] |
| 04:58 |
<mutante> |
install1002 - restarted isc-dhcpd |
[production] |
| 04:39 |
<mutante> |
phab1001 - rebooting for BIOS config change |
[production] |
| 02:06 |
<mutante> |
re-enabling puppet on phab1003 and phab1001.. switching active_server for puppet |
[production] |
| 01:55 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=phab1001-vcs.eqiad.wmnet |
[production] |
| 01:46 |
<mutante> |
switching phab-vcs in conftool-data from phab1003 to phab1001, running puppet on conf* |
[production] |
| 01:45 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=phab1003-vcs.eqiad.wmnet |
[production] |
| 01:40 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=phab1003-vcs.eqiad.wmnet |
[production] |
| 01:37 |
<twentyafterfour> |
re-enable phabricator writes (disable cluster.read-only) |
[production] |
| 01:33 |
<twentyafterfour> |
phab1001.eqiad.wmnet : sudo chown root.www-data /srv/phab/phabricator/conf/local/www.json |
[production] |
| 01:29 |
<mutante> |
phabricator currently under maintenance - db connection error is known |
[production] |
| 01:20 |
<mutante> |
running puppet on cp-eqiad |
[production] |
| 00:49 |
<ejegg> |
changed donations queue consumer and thank you mailer to use 3 minute cycles |
[production] |
| 00:41 |
<twentyafterfour> |
switching phabricator to read-only mode |
[production] |
| 00:40 |
<reedy@deploy1001> |
Synchronized php-1.35.0-wmf.8/skins/Vector/includes/templates/SearchComponent.mustache: I9776a3c355081dc5fec7753edf256f55dfe6045b (duration: 01m 01s) |
[production] |
|
2019-12-03
§
|
| 23:47 |
<volans> |
re-enabled meta-monitoring crontabs on wikitech-static after cleanup, reboot and fix wikitech-static's import errors |
[production] |
| 22:59 |
<volans> |
apt-get dist-upgrade and reboot of wikitech-static host |
[production] |
| 22:36 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Remove settings for closed wikis T231178 (duration: 01m 01s) |
[production] |
| 22:34 |
<volans> |
disabled temporarily icinga meta-monitoring (disk full on the wikitech-static host) |
[production] |
| 22:34 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Enable the Wikisource extension on frwikisource T239731 (duration: 01m 00s) |
[production] |
| 22:22 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Read wmgDoNotRedirectOnSearchMatch to decide to enable auto-redirect search result change T235263 (duration: 01m 00s) |
[production] |
| 22:21 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Set wmgDoNotRedirectOnSearchMatch, default off, on for Test Commons T235263 (duration: 01m 01s) |
[production] |
| 22:03 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Set wgXmlDumpSchemaVersion to 0.1.0 everywhere T238921 T174031 (duration: 01m 03s) |
[production] |
| 21:40 |
<eileen> |
civicrm revision changed from 26b788378e to 0f51030071, config revision is 17b6730a72 - includes 3 possible performance improvements - logging reduction, cache a query result & cache file existence |
[production] |
| 21:38 |
<volker-e@deploy1001> |
Finished deploy [design/style-guide@02a92f7]: Deploy design/style-guide: (duration: 00m 07s) |
[production] |
| 21:38 |
<volker-e@deploy1001> |
Started deploy [design/style-guide@02a92f7]: Deploy design/style-guide: |
[production] |
| 21:09 |
<sbassett> |
Deployed security patch for T238768 to wmf.8 |
[production] |
| 21:03 |
<sbassett> |
Deployed security patch for T238768 to wmf.5 |
[production] |
| 20:43 |
<mutante> |
mw2259 - did not come back from reboot after reimage, also mgmt not reachable (T239054) |
[production] |
| 20:38 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2257.codfw.wmnet |
[production] |
| 20:38 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2256.codfw.wmnet |
[production] |
| 20:38 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2258.codfw.wmnet |
[production] |
| 20:17 |
<bblack@cumin1001> |
conftool action : set/pooled=yes; selector: name=dns[12]002.wikimedia.org |
[production] |
| 20:00 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@c21a1ca]: Bump preq version for better logging around MW API timeouts (duration: 05m 46s) |
[production] |
| 19:54 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@c21a1ca]: Bump preq version for better logging around MW API timeouts |
[production] |
| 19:53 |
<ejegg> |
shifted 20 more sec / cycle from donations QC to thank you mailer |
[production] |
| 19:41 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |