|
2019-11-12
ยง
|
| 16:40 |
<ema@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:28 |
<XioNoX> |
setup bgp session from cr2-codfw to multihop RIS collector - T106056 |
[production] |
| 16:21 |
<XioNoX> |
reboot scs-c1-eqiad.mgmt.eqiad.wmnet - T238036 |
[production] |
| 16:09 |
<ema> |
depool cp3052 and observe performance impact T238085 before reimaging as text_ats T227432 |
[production] |
| 15:49 |
<marostegui> |
Deploy schema change on db1102:3315 T233135 T234066 |
[production] |
| 15:45 |
<mholloway-shell@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/MachineVision: Fixes and tweaks for initial rollout (duration: 00m 53s) |
[production] |
| 15:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1097:3315 for a schema change T233135 T234066', diff saved to https://phabricator.wikimedia.org/P9600 and previous config saved to /var/cache/conftool/dbconfig/20191112-154127-marostegui.json |
[production] |
| 15:24 |
<otto@puppetmaster1001> |
conftool action : set/pooled=true; selector: dnsdisc=schema |
[production] |
| 14:46 |
<bblack> |
cpNNNN (all caches): remove stale outputs from transient ocsp failures ( /var/cache/ocsp/update-ocsp-*.tmp ) |
[production] |
| 14:41 |
<ema> |
cp4022: trafficserver (8.0.5-1wm10) and fifo-log-demux (0.6) upgrade and restart |
[production] |
| 14:38 |
<ema@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=cp4021.ulsfo.wmnet,service=nginx |
[production] |
| 14:35 |
<ema> |
cp4021: ats-tls-restart to see if https://gerrit.wikimedia.org/r/550475 fixed the script |
[production] |
| 14:16 |
<Jeff_Green> |
authdns-update to deploy fundraising-read.wmnet service cname adjustment |
[production] |
| 14:01 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Revert "Set all of wikidata for write both for term store" (duration: 00m 52s) |
[production] |
| 12:57 |
<godog> |
refresh kibana field list |
[production] |
| 12:46 |
<gehel> |
repool wdqs1004 |
[production] |
| 12:37 |
<Amir1> |
ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildPropertyTerms.php --wiki=wikidatawiki --batch-size 100 (T237984) |
[production] |
| 12:19 |
<onimisionipe> |
restarting blazegraph on wdqs1005 |
[production] |
| 12:11 |
<effie> |
Reimage mwdebug1002 - T214734 |
[production] |
| 11:47 |
<Amir1> |
EU SWAT is done |
[production] |
| 11:47 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.4/extensions/Wikibase: Wikibase term store error reduction, [[gerrit:550441|Do not catch DBError in ReplicaMasterAwareRecordIdsAcquirer.]] (T236466) (duration: 00m 56s) |
[production] |
| 11:44 |
<effie> |
Upgrade wtp* to 7.2.24-1 with elegance and restart php-fpm - T237239 |
[production] |
| 11:20 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:548584|Set all of wikidata for write both for term store (T225055)]] (duration: 00m 52s) |
[production] |
| 11:05 |
<urbanecm@deploy1001> |
Synchronized wmf-config/CommonSettings.php: SECURITY: Dont allow Wikimedia sysops to see who had 2FA disabled (duration: 00m 53s) |
[production] |
| 10:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1083', diff saved to https://phabricator.wikimedia.org/P9599 and previous config saved to /var/cache/conftool/dbconfig/20191112-104400-marostegui.json |
[production] |
| 10:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'More traffic to db1083', diff saved to https://phabricator.wikimedia.org/P9598 and previous config saved to /var/cache/conftool/dbconfig/20191112-103641-marostegui.json |
[production] |
| 10:35 |
<onimisionipe> |
resetting cronfile on wdqs hosts |
[production] |
| 10:33 |
<marostegui> |
Drop labtestwiki database from m5 master db1133 - T236010 |
[production] |
| 10:30 |
<marostegui> |
Deploy schema change on dbstore1003:3315 |
[production] |
| 10:06 |
<ema> |
repool cp3065, nothing interesting in kern.log and SEL T238032 |
[production] |
| 09:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'More traffic to db1083', diff saved to https://phabricator.wikimedia.org/P9596 and previous config saved to /var/cache/conftool/dbconfig/20191112-095221-marostegui.json |
[production] |
| 09:42 |
<marostegui> |
Remove privileges for labtestwiki on m5 - T236010 |
[production] |
| 09:27 |
<gehel> |
restarting blazegraph on wdqs1004 |
[production] |
| 09:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1083', diff saved to https://phabricator.wikimedia.org/P9595 and previous config saved to /var/cache/conftool/dbconfig/20191112-091706-marostegui.json |
[production] |
| 09:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1083 for mariadb upgrade to 10.1.39 - T234800', diff saved to https://phabricator.wikimedia.org/P9594 and previous config saved to /var/cache/conftool/dbconfig/20191112-091158-marostegui.json |
[production] |
| 09:11 |
<marostegui> |
Upgrade mariadb to 10.1.39 on db1083 (candidate master for s1) |
[production] |
| 08:56 |
<moritzm> |
restarting archiva to pick up Java security updates |
[production] |
| 08:44 |
<volker-e@deploy1001> |
Finished deploy [design/style-guide@3de6820]: Deploy design/style-guide: (duration: 00m 06s) |
[production] |
| 08:44 |
<volker-e@deploy1001> |
Started deploy [design/style-guide@3de6820]: Deploy design/style-guide: |
[production] |
| 08:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1083', diff saved to https://phabricator.wikimedia.org/P9593 and previous config saved to /var/cache/conftool/dbconfig/20191112-083720-marostegui.json |
[production] |
| 08:37 |
<gehel> |
depool wdqs1004 to investigate update lag |
[production] |
| 08:35 |
<moritzm> |
installing poppler security updates |
[production] |
| 08:24 |
<volker-e@deploy1001> |
Finished deploy [design/style-guide@b926b95]: Deploy design/style-guide: (duration: 00m 07s) |
[production] |
| 08:24 |
<volker-e@deploy1001> |
Started deploy [design/style-guide@b926b95]: Deploy design/style-guide: |
[production] |
| 08:15 |
<moritzm> |
installing curl security updates |
[production] |
| 08:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase traffic to db1083', diff saved to https://phabricator.wikimedia.org/P9592 and previous config saved to /var/cache/conftool/dbconfig/20191112-081322-marostegui.json |
[production] |
| 07:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase traffic to db1083', diff saved to https://phabricator.wikimedia.org/P9591 and previous config saved to /var/cache/conftool/dbconfig/20191112-074006-marostegui.json |
[production] |
| 07:36 |
<elukey> |
remove /etc/logrotate.d/wdqs_autodeployment_log from wdqs1009 (not in puppet anymore and causing cronspam) |
[production] |
| 07:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1083 after kernel upgrade', diff saved to https://phabricator.wikimedia.org/P9590 and previous config saved to /var/cache/conftool/dbconfig/20191112-072823-marostegui.json |
[production] |
| 07:10 |
<marostegui> |
Upgrade kernel on db1083 (s1 candidate master) |
[production] |