|
2020-01-16
§
|
| 11:27 |
<akosiaris> |
delete etcd100{4,5,6} from netbox. T239835 |
[production] |
| 11:27 |
<akosiaris> |
delete etcd100{4,5,6} from ganeti01.svc.eqiad.wmnet. T239835 |
[production] |
| 11:22 |
<elukey> |
import packages in stretch-wikimedia's thirdparty/bigtop14 component |
[production] |
| 11:20 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 11:18 |
<volans> |
uploaded spicerack_0.0.29-1_amd64.deb to apt.wikimedia.org stretch-wikimedia |
[production] |
| 11:17 |
<vgutierrez> |
restarting pybal on lvs5001 (high-traffic1) - T242321 |
[production] |
| 11:16 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 11:16 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) |
[production] |
| 11:16 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 11:13 |
<vgutierrez> |
restarting pybal on lvs5003 (secondary LVS) - T242321 |
[production] |
| 11:10 |
<vgutierrez@puppetmaster1001> |
conftool action : set/pooled=yes:weight=1; selector: service=nginx,name=ncredir5002.eqsin.wmnet |
[production] |
| 11:10 |
<vgutierrez@puppetmaster1001> |
conftool action : set/pooled=yes:weight=1; selector: service=nginx,name=ncredir5001.eqsin.wmnet |
[production] |
| 09:31 |
<Amir1> |
"delete from wikidatawiki.wb_terms where term_full_entity_id like 'P%'" (T219301 T225054) |
[releng] |
| 09:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1100', diff saved to https://phabricator.wikimedia.org/P10171 and previous config saved to /var/cache/conftool/dbconfig/20200116-092409-root.json |
[production] |
| 09:18 |
<effie> |
restart php-fpm on mwmaint2001.codfw.wmnet,mwmaint1002.eqiad.wmnet,scandium.eqiad.wmnet |
[production] |
| 09:16 |
<effie> |
Updgrade mwmaint2001.codfw.wmnet,mwmaint1002.eqiad.wmnet,scandium.eqiad.wmnet, to php 7.2.26 - T241222 |
[production] |
| 09:09 |
<effie> |
restart php-fpm on cloudweb2001-dev.wikimedia.org,labweb[1001-1002].wikimedia.org |
[production] |
| 09:02 |
<effie> |
Updgrade cloudweb2001-dev.wikimedia.org,labweb[1001-1002].wikimedia.org to php 7.2.26 - T241222 |
[production] |
| 08:55 |
<ema> |
cp3063: ats-backend-restart to clear things up after traffic_server crash T242952 |
[production] |
| 08:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1110', diff saved to https://phabricator.wikimedia.org/P10170 and previous config saved to /var/cache/conftool/dbconfig/20200116-085047-marostegui.json |
[production] |
| 08:39 |
<effie> |
Upgrade deploy*, snapshot* to php 7.2.26 - T241222 |
[production] |
| 08:27 |
<moritzm> |
installing OpenSSL security updates on Parsoid hosts |
[production] |
| 08:20 |
<XioNoX> |
reject RPKI invalids in eqord/eqiad - T220669 |
[production] |
| 08:05 |
<_joe_> |
deleting mediawiki-core docker images from september 2019 from the registry, T242775 |
[production] |
| 07:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1110', diff saved to https://phabricator.wikimedia.org/P10169 and previous config saved to /var/cache/conftool/dbconfig/20200116-073012-marostegui.json |
[production] |
| 07:22 |
<marostegui> |
Upgrade db1110 |
[production] |
| 07:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1110', diff saved to https://phabricator.wikimedia.org/P10168 and previous config saved to /var/cache/conftool/dbconfig/20200116-072219-marostegui.json |
[production] |
| 06:58 |
<marostegui> |
stop db1107 and db1080 replication in sync |
[production] |
| 06:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1080', diff saved to https://phabricator.wikimedia.org/P10166 and previous config saved to /var/cache/conftool/dbconfig/20200116-065505-marostegui.json |
[production] |
| 02:46 |
<Krinkle> |
krinkle@mwmaint1002 Change code_repo.repo_viewvc from 'http://svn.wikimedia.org/viewvc/pywikipedia' to '' for repo_id 2 (pywikipedia) for. Ref 2162cf2fc46cfe. |
[production] |
| 02:35 |
<Krinkle> |
krinkle@mwmaint1002 Change code_repo.repo_viewvc from 'https://svn.wikimedia.org/viewvc/mediawiki' to '' for 'MediaWiki' repo_name. Ref 2162cf2fc46cfe, T205361. |
[production] |
| 00:53 |
<mutante> |
deploy-1002 - become 'trebuchet' user and ssh to phabricator scap targets. to fix ssh host key verification issue on first deploy |
[devtools] |
| 00:40 |
<bstorm_> |
set max_connections on db1133 (m5-master) back to 500 since the neutron connections seem fairly stable now T242817 |
[production] |
| 00:30 |
<mutante> |
deploy-1002 live hack /srv/deployment/phabricator/deployment/scap/phabricator-targets and replace prod server with cloud instances; scap deploy in phabricator repo |
[devtools] |
| 00:23 |
<catrope@deploy1001> |
Synchronized static/images/project-logos/: Restore pre-censorship trwiki logos (T242932) (duration: 01m 05s) |
[production] |
| 00:17 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: GrowthExperiments: Enable topics for suggested edits on testwiki (duration: 01m 04s) |
[production] |
|
2020-01-15
§
|
| 23:51 |
<paladox> |
deploy-1002 rm -rf /srv/deployment |
[devtools] |
| 23:44 |
<mutante> |
deploy-1002 sudo git init in /srv/deployment ; scap deploy --init (now fails with 'fatal: Not a valid object name HEAD') |
[devtools] |
| 23:42 |
<mutante> |
deploy-1002 mkdir /srv/deployments/.git ; chown trebuchet:wikidev .git ; manually run "scap deploy --init" as trebuchet user in an attempt to fix initial puppet run on deployment_server |
[devtools] |
| 23:00 |
<Reedy> |
deleted `contint2001:/srv/zuul/git/mediawiki/services/change-propagation` as the repo seems to be corrupted (git remote was integration/docroot) like T134062 |
[releng] |
| 22:43 |
<Reedy> |
deleted `contint1001:/srv/zuul/git/mediawiki/services/change-propagation` as the repo seems to be corrupted (git remote was integration/docroot) like T134062 |
[releng] |
| 22:40 |
<mutante> |
phabricator - disabling 'bzimport' user (T242860) |
[production] |
| 21:03 |
<jforrester@deploy1001> |
Synchronized php-1.35.0-wmf.14/languages/messages/MessagesMrj.php: Fix fallbacks of mrj (Hill Mari) T242409 T242796 (duration: 01m 05s) |
[production] |
| 20:56 |
<mutante> |
deleting instance gerrit-mysql (used be icinga2 for ores, but ores has prod icinga checks at least) T236569 |
[git] |
| 20:47 |
<mutante> |
gerrit - adding Zoranzoki to members of extension-GoogleAdSense (endorsed by extension owner Siebrand) (T241509) |
[production] |
| 20:28 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Touched IS.php for sync (duration: 01m 05s) |
[production] |
| 20:27 |
<jforrester@deploy1001> |
sync-file aborted: Enable partial blocks on last wiki, (duration: 00m 01s) |
[production] |
| 20:17 |
<krinkle@deploy1001> |
Synchronized php-1.35.0-wmf.15/extensions/MultimediaViewer/resources/: T229484 (duration: 01m 06s) |
[production] |
| 19:57 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Enable partial blocks on last wiki, Commons T242570 (duration: 01m 03s) |
[production] |
| 19:54 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Enable banner for wikis that recently opted in to partial blocks T240300 T242570 T242569 (duration: 01m 05s) |
[production] |