|
2019-04-10
§
|
| 09:59 |
<moritzm> |
upgrading labweb hosts (wikitech) to HHVM 3.18.5+dfsg-1+wmf8+deb9u2 and wikidiff 1.8.1 (T203069) |
[production] |
| 09:58 |
<arturo> |
force deleted job 871945 because it was stucked (trying to depool exec node for T218126) |
[tools.citationhunt] |
| 09:57 |
<arturo> |
force deleted job 853968 because it was stucked (trying to depool exec node for T218126) |
[tools.wmcounter] |
| 09:56 |
<arturo> |
force deleted job 853139 because it was stucked (trying to depool exec node for T218126) |
[tools.quickstatements] |
| 09:51 |
<akosiaris> |
upgrade kubernetes-node on kubestage1001 (staging cluster) to 1.12.7-1 T220405 |
[production] |
| 09:50 |
<moritzm> |
upgrading snapshot hosts to HHVM 3.18.5+dfsg-1+wmf8+deb9u2 and wikidiff 1.8.1 (T203069) |
[production] |
| 09:40 |
<arturo> |
T218126 hard reboot tools-sgeexec-0918 |
[tools] |
| 09:40 |
<akosiaris> |
upgrade kubernetes-master on neon (staging cluster) to 1.12.7-1 T220405 |
[production] |
| 09:40 |
<akosiaris> |
upgrade kubernetes-master on neon (staging cluster) to 1.12.7-1 |
[production] |
| 09:27 |
<arturo> |
T218126 hard reboot tools-sgeexec-0932 |
[tools] |
| 09:26 |
<arturo> |
T218216 hard reboot tools-sgeexec-0932 |
[tools] |
| 09:05 |
<moritzm> |
upgrading job runners mw1299-mw1311 to HHVM 3.18.5+dfsg-1+wmf8+deb9u2 and wikidiff 1.8.1 (T203069) |
[production] |
| 09:04 |
<arturo> |
T218216 add `profile::ldap::client::labs::client_stack: sssd` to prefix puppet for sge-exec nodes |
[tools] |
| 09:03 |
<arturo> |
T218216 do a controlled rollover of sssd, depooling sgeexec nodes, reboot and repool |
[tools] |
| 08:56 |
<elukey> |
restart druid-broker on druid100[4-6] - stuck after attempt datasource delete action |
[production] |
| 08:46 |
<godog> |
roll-restart swift frontends - T214289 |
[production] |
| 08:39 |
<arturo> |
T218216 disable puppet in all tools-sgeexec-XXXX nodes for controlled sssd rollout |
[tools] |
| 08:36 |
<elukey> |
update thirdparty/cloudera packages to cdh 5.16.1 for jessie/stretch-wikimedia - T218343 |
[production] |
| 08:26 |
<onimisionipe@deploy1001> |
Finished deploy [kartotherian/deploy@f7518bb] (stretch): Insert maps2003 into stretch environment (duration: 00m 22s) |
[production] |
| 08:26 |
<onimisionipe@deploy1001> |
Started deploy [kartotherian/deploy@f7518bb] (stretch): Insert maps2003 into stretch environment |
[production] |
| 08:12 |
<gilles> |
T220265 foreachwiki extensions/WikimediaMaintenance/filebackend/setZoneAccess.php --backend local-multiwrite |
[production] |
| 07:22 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@efd5bd5]: Revert "Bifurcate imageinfo queries to improve performance" (T220574) (duration: 04m 05s) |
[production] |
| 07:18 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@efd5bd5]: Revert "Bifurcate imageinfo queries to improve performance" (T220574) |
[production] |
| 07:12 |
<onimisionipe> |
depooling maps200[34] to increase cassandra replication factor - T198622 |
[production] |
| 07:09 |
<jijiki> |
Rolling restart thumbor service |
[production] |
| 07:08 |
<jijiki> |
Upgrading Thumbor servers to python-thumbor-wikimedia to 2.4-1+deb9u1 |
[production] |
| 06:59 |
<marostegui> |
Deploy schema change on x1 master, with replication, lag will happen on x1 T217453 |
[production] |
| 06:59 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool x1 slaves T217453 (duration: 01m 13s) |
[production] |
| 05:52 |
<_joe_> |
setting both mwdebug200{1,2} to pooled = inactive to remove them from scap dsh list and allow deployments, T219989 |
[production] |
| 05:12 |
<_joe_> |
same on mwdebug2001 |
[production] |
| 05:11 |
<Cam11598> |
10:11:08 PM <ChanServ> Flags +AV were set on DannyS712 in #cvn-wp-en. |
[cvn] |
| 05:08 |
<_joe_> |
removing hhvm cache on mwdebug2002 |
[production] |
| 00:37 |
<Krinkle> |
last scap sync-file failed to mwdebug2002.codfw and mwdebug2001.codfw due to insufficient disk space |
[production] |
| 00:32 |
<andrewbogott> |
migrating tools-worker-1022, 1023, 1025, 1026 to eqiad1-r |
[tools] |
| 00:20 |
<krinkle@deploy1001> |
Synchronized php-1.33.0-wmf.25/resources/src/startup/: I3b9f1a13379a / Ie9db60e417cca (duration: 01m 01s) |
[production] |
|
2019-04-09
§
|
| 23:32 |
<bd808> |
Updated crontab to correct day of week (Wed==3) and hour (German summer time) |
[tools.taim-bot] |
| 23:26 |
<bd808> |
Test |
[tools.stashbot] |
| 23:14 |
<twentyafterfour@deploy1001> |
Pruned MediaWiki: 1.33.0-wmf.17 [keeping static files] (duration: 06m 03s) |
[production] |
| 23:08 |
<Krinkle> |
Reloading Zuul to deploy https://phabricator.wikimedia.org/T220561 |
[releng] |
| 22:31 |
<twentyafterfour@deploy1001> |
Finished scap: testwikis wikis to 1.33.0-wmf.25 refs T206679 (duration: 39m 59s) |
[production] |
| 22:19 |
<chaomodus> |
uploaded python-pynetbox to apt.wikimedia.org/stretch-wikimedia (T217072) |
[production] |
| 22:13 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@c0a2977]: Bring RB on restbase20(19|20) up to date - T208087 (duration: 02m 32s) |
[production] |
| 22:11 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@c0a2977]: Bring RB on restbase20(19|20) up to date - T208087 |
[production] |
| 22:04 |
<bstorm_> |
added the new region on port 80 to the elasticsearch security group for stashbot |
[tools] |
| 20:43 |
<andrewbogott> |
moving tools-worker-1018, 1019, 1020, 1021 to eqiad1-r |
[tools] |
| 20:04 |
<andrewbogott> |
moving tools-k8s-etcd-03 to eqiad1-r |
[tools] |
| 19:54 |
<andrewbogott> |
moving tools-flannel-etcd-02 to eqiad1-r |
[tools] |
| 19:46 |
<herron> |
added myself to ldap group cn=archiva-deployers,ou=groups,dc=wikimedia,dc=org |
[production] |
| 19:36 |
<bstorm_> |
T220543 stopped webservice because it runs up load on any node it runs on in its current state and needs fixing before it is run |
[tools.montage-beta] |
| 19:28 |
<bstorm_> |
restarted webservice because it was hung in uninterruptible sleep and not working--causing very high load as well |
[tools.montage-beta] |