|
2020-01-31
§
|
| 18:14 |
<bblack> |
repool cp4029 |
[production] |
| 18:13 |
<bblack> |
restarted ats-tls and varnish-fe on cp4029 |
[production] |
| 18:05 |
<bblack> |
depool varnish-fe on cp4029 |
[production] |
| 18:03 |
<bblack> |
depool ats-tls on cp4029 |
[production] |
| 16:59 |
<marostegui> |
Re-enable notifications on the dbstore1005:3318 check T243871 |
[production] |
| 09:18 |
<addshore> |
addshore@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildPropertyTerms.php --wiki=wikidatawiki --sleep 4 --batch-size=25 # In a screen for T219301 |
[production] |
| 03:22 |
<mutante> |
powercycling crashed cp3063 |
[production] |
| 01:09 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@322ee4c]: Update mobileapps to 3eec28d (duration: 06m 53s) |
[production] |
| 01:02 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@322ee4c]: Update mobileapps to 3eec28d |
[production] |
| 00:41 |
<mutante> |
contint1001/contint2001 - upgrading jenkins to 2.219 |
[production] |
| 00:36 |
<mutante> |
releases2001: upgrading jenkins to 2.219; install1002: import jenkins 2.219 into jessie-wikimedia APT repo |
[production] |
| 00:31 |
<mutante> |
importing jenkins 2.219 to stretch-wikimedia APT repo; releases1001: upgrading jenkins to 2.219 |
[production] |
|
2020-01-30
§
|
| 19:37 |
<mutante> |
copying /var/log/apache2 to /root on all eqiad mw appservers to preserve logs |
[production] |
| 18:07 |
<vgutierrez> |
depool cp4032 and perform a rolling restart of varnish-fe at cp4027-cp4031 - T243634 |
[production] |
| 17:51 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.16/extensions/Wikibase/lib/includes/Store/Sql/Terms/FingerprintableEntityTermStoreTrait.php: wbterms: Fix incorrect deletion of rows in findActuallyUnusedTermIds (T243944) (duration: 01m 06s) |
[production] |
| 17:49 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.16/extensions/Wikibase/repo/maintenance/rebuildItemTerms.php: wbterms: Write only to the new term store in rebuildItemTerms (T243944) (duration: 01m 09s) |
[production] |
| 17:03 |
<vgutierrez> |
repooling cp4032 - T243634 |
[production] |
| 17:02 |
<vgutierrez> |
restarting varnish-frontend on cp4031 before it crashes - T243634 |
[production] |
| 16:26 |
<vgutierrez> |
manually refreshing OCSP stapling response for non-canonical-redirects-3 - T243948 |
[production] |
| 12:22 |
<arturo> |
add prometheus 2.7.1+ds-3+k8s+buster to buster-wikimedia T238096 (basically a rebuild from stretch) |
[production] |
| 06:23 |
<vgutierrez> |
restarting varnish-frontend on cp4030 before it crashes - T243634 |
[production] |
| 06:21 |
<vgutierrez> |
depool cp4032 - T243634 |
[production] |
| 05:12 |
<vgutierrez> |
restarting varnish-frontend and repooling cp4029 - T243634 |
[production] |
| 05:00 |
<vgutierrez> |
depooling cp4029 |
[production] |
|
2020-01-28
§
|
| 23:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1097:3314 T239453', diff saved to https://phabricator.wikimedia.org/P10287 and previous config saved to /var/cache/conftool/dbconfig/20200128-235336-marostegui.json |
[production] |
| 23:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1097:3314 T239453', diff saved to https://phabricator.wikimedia.org/P10286 and previous config saved to /var/cache/conftool/dbconfig/20200128-234601-marostegui.json |
[production] |
| 23:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Start repooling db1084 with its original weight', diff saved to https://phabricator.wikimedia.org/P10285 and previous config saved to /var/cache/conftool/dbconfig/20200128-234219-marostegui.json |
[production] |
| 23:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1121 T232446', diff saved to https://phabricator.wikimedia.org/P10284 and previous config saved to /var/cache/conftool/dbconfig/20200128-234037-marostegui.json |
[production] |
| 15:06 |
<addshore> |
Start addshore@mwmaint1002:~$ ./T219123.sh # Taking over from @ladsgroup for T219123 |
[production] |
| 09:59 |
<effie> |
rolling restart mobileapps in codfw |
[production] |
| 02:05 |
<mutante> |
gerrit1002 - gzipping a bunch of /var/log/gerrit/ log files (T243808) |
[production] |
|
2020-01-27
§
|
| 23:40 |
<eileen> |
civicrm revision changed from fbd5c35fb0 to ac730a6bcb, config revision is 837b9d0703 |
[production] |
| 23:10 |
<vgutierrez> |
rolling restart of varnish-frontend in cp4026 and cp4027 |
[production] |
| 23:06 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 23:06 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 23:01 |
<_joe_> |
restart apache on gerrit |
[production] |
| 22:58 |
<vgutierrez> |
restarting gerrit service |
[production] |
| 22:01 |
<vgutierrez> |
restarting varnish-fe on cp4028 |
[production] |
| 19:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2085:3311 - T239453', diff saved to https://phabricator.wikimedia.org/P10277 and previous config saved to /var/cache/conftool/dbconfig/20200127-191614-marostegui.json |
[production] |
| 19:15 |
<marostegui> |
Remove partitions from db2085 enwiki - T239453 |
[production] |
| 13:58 |
<vgutierrez> |
repooling cp4030 - T243634 |
[production] |