|
2019-11-12
ยง
|
| 19:03 |
<arlolra@deploy1001> |
Finished deploy [parsoid/deploy@f516018]: Updating Parsoid to 6a0a708 (duration: 10m 09s) |
[production] |
| 18:58 |
<mholloway-shell@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/MachineVision: Final fixes and tweaks for testing (duration: 00m 53s) |
[production] |
| 18:53 |
<arlolra@deploy1001> |
Started deploy [parsoid/deploy@f516018]: Updating Parsoid to 6a0a708 |
[production] |
| 18:39 |
<ejegg> |
re-enabled Omnimail and contact de-duplication jobs |
[production] |
| 18:20 |
<Urbanecm> |
Morning SWAT done |
[production] |
| 18:18 |
<Urbanecm> |
Deploy security patch for T237887 |
[production] |
| 18:15 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 130ef87: Add right "abusefilter-log-private" to usergroup "rollbacker" at ptwiki (T237830) (duration: 00m 53s) |
[production] |
| 18:08 |
<XioNoX> |
push pfw change to add recdns anycast IP |
[production] |
| 17:33 |
<XioNoX> |
update fasw-c-eqiad to match current standard (ntp/users/rootpw/lldp) |
[production] |
| 17:22 |
<XioNoX> |
update fasw-c-codfw to match current standard (ntp/users/rootpw/lldp) |
[production] |
| 17:03 |
<ema> |
pool cp3052 with ATS backend T238085 |
[production] |
| 17:03 |
<ema> |
pool cp3052 with ATS backend T227432 |
[production] |
| 16:53 |
<bblack> |
cpNNNN (all cache nodes) - cumin manual removal of globalsign-2018 remnants (key, cert, ocsp config, ocsp output) |
[production] |
| 16:42 |
<ema@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:40 |
<ema@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:28 |
<XioNoX> |
setup bgp session from cr2-codfw to multihop RIS collector - T106056 |
[production] |
| 16:21 |
<XioNoX> |
reboot scs-c1-eqiad.mgmt.eqiad.wmnet - T238036 |
[production] |
| 16:09 |
<ema> |
depool cp3052 and observe performance impact T238085 before reimaging as text_ats T227432 |
[production] |
| 15:49 |
<marostegui> |
Deploy schema change on db1102:3315 T233135 T234066 |
[production] |
| 15:45 |
<mholloway-shell@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/MachineVision: Fixes and tweaks for initial rollout (duration: 00m 53s) |
[production] |
| 15:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1097:3315 for a schema change T233135 T234066', diff saved to https://phabricator.wikimedia.org/P9600 and previous config saved to /var/cache/conftool/dbconfig/20191112-154127-marostegui.json |
[production] |
| 15:24 |
<otto@puppetmaster1001> |
conftool action : set/pooled=true; selector: dnsdisc=schema |
[production] |
| 14:46 |
<bblack> |
cpNNNN (all caches): remove stale outputs from transient ocsp failures ( /var/cache/ocsp/update-ocsp-*.tmp ) |
[production] |
| 14:41 |
<ema> |
cp4022: trafficserver (8.0.5-1wm10) and fifo-log-demux (0.6) upgrade and restart |
[production] |
| 14:38 |
<ema@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=cp4021.ulsfo.wmnet,service=nginx |
[production] |
| 14:35 |
<ema> |
cp4021: ats-tls-restart to see if https://gerrit.wikimedia.org/r/550475 fixed the script |
[production] |
| 14:16 |
<Jeff_Green> |
authdns-update to deploy fundraising-read.wmnet service cname adjustment |
[production] |
| 14:01 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Revert "Set all of wikidata for write both for term store" (duration: 00m 52s) |
[production] |
| 12:57 |
<godog> |
refresh kibana field list |
[production] |
| 12:46 |
<gehel> |
repool wdqs1004 |
[production] |
| 12:37 |
<Amir1> |
ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildPropertyTerms.php --wiki=wikidatawiki --batch-size 100 (T237984) |
[production] |
| 12:19 |
<onimisionipe> |
restarting blazegraph on wdqs1005 |
[production] |
| 12:11 |
<effie> |
Reimage mwdebug1002 - T214734 |
[production] |
| 11:47 |
<Amir1> |
EU SWAT is done |
[production] |
| 11:47 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.4/extensions/Wikibase: Wikibase term store error reduction, [[gerrit:550441|Do not catch DBError in ReplicaMasterAwareRecordIdsAcquirer.]] (T236466) (duration: 00m 56s) |
[production] |
| 11:44 |
<effie> |
Upgrade wtp* to 7.2.24-1 with elegance and restart php-fpm - T237239 |
[production] |
| 11:20 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:548584|Set all of wikidata for write both for term store (T225055)]] (duration: 00m 52s) |
[production] |
| 11:05 |
<urbanecm@deploy1001> |
Synchronized wmf-config/CommonSettings.php: SECURITY: Dont allow Wikimedia sysops to see who had 2FA disabled (duration: 00m 53s) |
[production] |
| 10:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1083', diff saved to https://phabricator.wikimedia.org/P9599 and previous config saved to /var/cache/conftool/dbconfig/20191112-104400-marostegui.json |
[production] |
| 10:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'More traffic to db1083', diff saved to https://phabricator.wikimedia.org/P9598 and previous config saved to /var/cache/conftool/dbconfig/20191112-103641-marostegui.json |
[production] |
| 10:35 |
<onimisionipe> |
resetting cronfile on wdqs hosts |
[production] |
| 10:33 |
<marostegui> |
Drop labtestwiki database from m5 master db1133 - T236010 |
[production] |
| 10:30 |
<marostegui> |
Deploy schema change on dbstore1003:3315 |
[production] |
| 10:06 |
<ema> |
repool cp3065, nothing interesting in kern.log and SEL T238032 |
[production] |
| 09:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'More traffic to db1083', diff saved to https://phabricator.wikimedia.org/P9596 and previous config saved to /var/cache/conftool/dbconfig/20191112-095221-marostegui.json |
[production] |
| 09:42 |
<marostegui> |
Remove privileges for labtestwiki on m5 - T236010 |
[production] |
| 09:27 |
<gehel> |
restarting blazegraph on wdqs1004 |
[production] |
| 09:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1083', diff saved to https://phabricator.wikimedia.org/P9595 and previous config saved to /var/cache/conftool/dbconfig/20191112-091706-marostegui.json |
[production] |
| 09:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1083 for mariadb upgrade to 10.1.39 - T234800', diff saved to https://phabricator.wikimedia.org/P9594 and previous config saved to /var/cache/conftool/dbconfig/20191112-091158-marostegui.json |
[production] |
| 09:11 |
<marostegui> |
Upgrade mariadb to 10.1.39 on db1083 (candidate master for s1) |
[production] |