|
2019-04-03
§
|
| 12:49 |
<gehel@cumin2001> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=99) |
[production] |
| 12:45 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 12:42 |
<arturo> |
T219626 reimaging cloudcontrol2001-dev |
[production] |
| 12:31 |
<mutante> |
restarting gerrit service to apply change 498431 |
[production] |
| 11:25 |
<Amir1> |
EU SWAT is done |
[production] |
| 11:16 |
<jbond42> |
rolling security updates for apache |
[production] |
| 10:29 |
<mutante> |
planet1001/2001 - apt autoremove un-required packages |
[production] |
| 10:27 |
<mutante> |
planet1001/2001 - upgrade apache2, openssh, locales, rsyslog .. |
[production] |
| 10:25 |
<arturo> |
updating puppet compiler facts |
[production] |
| 10:19 |
<volans> |
upgraded spicerack to 0.0.21 on cumin[12]001 |
[production] |
| 10:17 |
<volans> |
uploaded spicerack_0.0.21-1_amd64.deb to apt.wikimedia.org stretch-wikimedia |
[production] |
| 09:56 |
<marostegui> |
Alter empty job table on s6 primary master - T219887 |
[production] |
| 09:55 |
<moritzm> |
upgrading beta to hhvm wikidiff 1.8.1 (T203069) |
[production] |
| 09:54 |
<mutante> |
running mysql select queries on m3-slave to get data from phabricator conpherence as requested by andre |
[production] |
| 09:45 |
<moritzm> |
removed labtestnet2003.codfw.wmnet from debmonitor (T219776) |
[production] |
| 09:29 |
<ema> |
cp-ats-codfw: test ATS rolling restart T213263 |
[production] |
| 09:27 |
<marostegui> |
Drop wikishared.wikimedia_editor_tasks_entity_description_exists table from x1 T219963 |
[production] |
| 09:27 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool s8 sanitarium master (duration: 00m 56s) |
[production] |
| 09:26 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool s8 sanitarium master (duration: 01m 00s) |
[production] |
| 08:35 |
<jynus> |
merging change on network constants (firewall operation) |
[production] |
| 08:23 |
<marostegui> |
Restart mysql on sanitarium hosts db1124 db1125 db2094 db2095 - T218302 |
[production] |
| 08:18 |
<marostegui> |
Stop replication on db2082 and db1087 (s8 sanitarium masters) T218302 |
[production] |
| 08:17 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool s8 sanitarium master (duration: 00m 57s) |
[production] |
| 08:16 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool s8 sanitarium master (duration: 00m 58s) |
[production] |
| 08:09 |
<moritzm> |
installing new apache packages on mmw1261 |
[production] |
| 07:53 |
<gilles@deploy1001> |
Synchronized php-1.33.0-wmf.24/includes/media/ThumbnailImage.php: T216499 Only apply high priority hint half the time (duration: 00m 58s) |
[production] |
| 07:51 |
<moritzm> |
installing new apache packages on mwdebug |
[production] |
| 07:42 |
<marostegui> |
Reboot db1115 - tendril and dbtree will be down |
[production] |
| 07:40 |
<marostegui> |
DIsable event scheduler on db1115 before restarting - tendril is stuck |
[production] |
| 07:26 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1120 T219493 (duration: 00m 57s) |
[production] |
| 07:25 |
<marostegui> |
Deploy schema change on db1073, labtestwiki - T219887 |
[production] |
| 07:09 |
<marostegui> |
Stop replication in sync on db1120 and db2034 (x1 codfw master) - T219493 |
[production] |
| 07:07 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1120 T219493 (duration: 01m 13s) |
[production] |
| 06:04 |
<_joe_> |
restart varnish backend on cp1085, causing unavailability |
[production] |
| 05:57 |
<marostegui> |
Fix data drifts on bnwikisource on x1 - T219493 |
[production] |
| 05:31 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool pc1007 (duration: 00m 59s) |
[production] |
| 05:23 |
<marostegui> |
Upgrade pc1007 |
[production] |
| 05:23 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool pc1007 for upgrade (duration: 01m 00s) |
[production] |
|
2019-04-02
§
|
| 23:44 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT enwiki: Restrict move-categorypages to +extendedmover/+sysop/+bot T219261 (duration: 00m 58s) |
[production] |
| 23:30 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT Add new WMCS IP range to wgRateLimitsExcludedIps T167432 (duration: 00m 57s) |
[production] |
| 23:27 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT Enable SandboxLink for rowiki T219855 (duration: 00m 56s) |
[production] |
| 23:22 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SDC: Add 'depicts' statements to search index on testcommons (duration: 00m 59s) |
[production] |
| 21:27 |
<andrewbogott> |
rebooting labservices1001 |
[production] |
| 21:16 |
<andrewbogott> |
rebooting labservices1002 |
[production] |
| 20:54 |
<andrewbogott> |
restarting pdns and pdns-recursor on labservices1001 and 1002 in hopes of getting those machines to act a bit less sluggish |
[production] |
| 20:23 |
<krinkle@deploy1001> |
Synchronized php-1.33.0-wmf.24/skins/Vector/includes/: I6e04b512d / T219864 (duration: 00m 59s) |
[production] |
| 20:20 |
<krinkle@deploy1001> |
Synchronized php-1.33.0-wmf.23/skins/Vector/includes/: I6e04b512d / T219864 (duration: 01m 00s) |
[production] |
| 20:16 |
<marxarelli> |
1.33.0-wmf.24 successfully deployed to group0. errors rates look normal (T206678) |
[production] |
| 20:07 |
<dduvall@deploy1001> |
rebuilt and synchronized wikiversions files: Group0 to 1.33.0-wmf.24 |
[production] |
| 19:57 |
<dduvall@deploy1001> |
Finished scap: testwiki to php-1.33.0-wmf.24 and rebuild l10n cache (duration: 44m 20s) |
[production] |