|
2021-01-07
§
|
| 14:32 |
<jayme> |
imported calico 3.17.0-2 to component/calico-future stretch-wikimedia |
[production] |
| 14:32 |
<jayme@deploy1001> |
helmfile [staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 14:08 |
<moritzm> |
installing sqlite3 security updates on buster |
[production] |
| 12:53 |
<dcaro> |
Taking osd.0 down on codfw ceph cluster to try the disk performance testing process (T271417) |
[admin] |
| 12:42 |
<_joe_> |
running puppet on logstash1007, elasticsearch oomkilled |
[production] |
| 12:24 |
<marostegui> |
Deploy schema change on s2 primary master with replication T270053 |
[production] |
| 12:21 |
<kart_> |
EU-Mid day backport window done. |
[production] |
| 12:12 |
<kartik@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:654734|Enable ContentTranslation in Tsonga Wikipedia as a default tool (T271204)]] (duration: 01m 09s) |
[production] |
| 12:02 |
<XioNoX> |
push "Allow specific flows from 172.16/12 to prod + default permit" - T209082 |
[production] |
| 11:35 |
<arturo> |
merging dmz_cidr change (T209082, T267779) |
[admin] |
| 11:24 |
<elukey> |
execute "sudo -u hdfs kerberos-run-command hdfs hdfs dfs -chmod -R o-rwx /wmf/data/event_sanitized" to fix some file permissions as well |
[analytics] |
| 11:16 |
<dcaro> |
removing custom mx hosts, as the global names are now resolvable again (T271322) |
[account-creation-assistance] |
| 10:51 |
<godog> |
swift eqiad-prod: add weight to ms-be106[0-3] - T268435 |
[production] |
| 10:36 |
<elukey> |
execute "sudo -u hdfs kerberos-run-command hdfs hdfs dfs -chmod -R o-rwx /wmf/data/event" on an-master1001 to fix some file permissions (an-launcher executed timers during the past hours without the new umask) - T270629 |
[analytics] |
| 10:06 |
<godog> |
bounce apache on prometheus codfw |
[production] |
| 09:50 |
<dcaro> |
Removing old recorset that has been moved to cloudinfra for mx-out02.wmflabs.org (T271322) |
[wmflabsdotorg] |
| 09:49 |
<dcaro> |
Added recordset for mx-out02.wmflabs.org (T271322) |
[cloudinfra] |
| 09:47 |
<dcaro> |
Removing old recorset that has been moved to cloudinfra for mx-out01.wmflabs.org (T271322) |
[wmflabsdotorg] |
| 09:46 |
<dcaro> |
Added recordset for mx-out01.wmflabs.org (T271322) |
[cloudinfra] |
| 09:37 |
<elukey> |
forced re-run of monitor_refine_event_failure_flags.service on an-launcher1002 to clear alerts |
[analytics] |
| 09:37 |
<moritzm> |
installing nodejs security updates on buster |
[production] |
| 09:27 |
<XioNoX> |
push new pfw policies - T271384 |
[production] |
| 09:17 |
<XioNoX> |
re-pool eqsin - T267544 |
[production] |
| 08:57 |
<XioNoX> |
re-enable BGP on cr2-eqsin - T267544 |
[production] |
| 08:26 |
<joal> |
Rerunning 4 failed refine jobs (mediawiki_cirrussearch_request, day=6/hour=20|21, day=7/hour=0|2) |
[analytics] |
| 08:14 |
<elukey> |
re-enable puppet on an-launcher1002 to apply new refine memory settings |
[analytics] |
| 08:14 |
<XioNoX> |
shutdown cr2-eqsin - T267544 |
[production] |
| 07:59 |
<elukey> |
re-enabling all oozie jobs previously suspended |
[analytics] |
| 07:54 |
<elukey> |
restart oozie on an-coord1001 |
[analytics] |
| 07:43 |
<moritzm> |
installing libxml2 security updates on buster |
[production] |
| 07:19 |
<XioNoX> |
depool eqsin for router replacement - T267544 |
[production] |
| 04:12 |
<andrew@deploy1001> |
Finished deploy [striker/deploy@e4db843]: update codfw1dev striker (duration: 00m 48s) |
[production] |
| 04:11 |
<andrew@deploy1001> |
Started deploy [striker/deploy@e4db843]: update codfw1dev striker |
[production] |
| 02:57 |
<andrew@deploy1001> |
Finished deploy [striker/deploy@e120c6c]: update codfw1dev striker (duration: 00m 05s) |
[production] |
| 02:57 |
<andrew@deploy1001> |
Started deploy [striker/deploy@e120c6c]: update codfw1dev striker |
[production] |
| 02:49 |
<andrew@deploy1001> |
Finished deploy [horizon/deploy@ce4c515]: update codfw1dev deploy from train-buster branch (duration: 01m 48s) |
[production] |
| 02:48 |
<andrew@deploy1001> |
Started deploy [horizon/deploy@ce4c515]: update codfw1dev deploy from train-buster branch |
[production] |
| 00:50 |
<eileen> |
civicrm revision changed from 95fefcb421 to 2df572bdcd, config revision is 603c0ff4a0 |
[production] |
| 00:44 |
<shdubsh> |
restart elasticsearch on logstash1011 - oom |
[production] |
|
2021-01-06
§
|
| 22:34 |
<sbassett> |
Deployed security patch for T270988 to wmf.22 |
[production] |
| 22:32 |
<sbassett> |
Deployed security patch for T270988 to wmf.25 |
[production] |
| 22:17 |
<legoktm> |
forced gerrit to replicate RequestTimeout to github (`ssh gerrit.wikimedia.org replication start mediawiki/libs/RequestTimeout --wait`) |
[production] |
| 21:59 |
<dwisehaupt> |
increased batch size for eoy_receipt to 4000 per run when frmx1001 is the primary mx |
[production] |
| 21:58 |
<dwisehaupt> |
process-control config revision is 603c0ff4a0 |
[production] |
| 21:25 |
<andrew@deploy1001> |
Finished deploy [horizon/deploy@b285acd]: update codfw1dev deploy from train-buster branch (duration: 09m 04s) |
[production] |
| 21:23 |
<jhuneidi@deploy1001> |
Synchronized php: group1 wikis to 1.36.0-wmf.25 refs T267418 (duration: 01m 04s) |
[production] |
| 21:22 |
<jhuneidi@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.25 refs T267418 |
[production] |
| 21:16 |
<andrew@deploy1001> |
Started deploy [horizon/deploy@b285acd]: update codfw1dev deploy from train-buster branch |
[production] |
| 21:16 |
<andrew@deploy1001> |
Finished deploy [horizon/deploy@b285acd]: update codfw1dev deploy from train-buster branch (duration: 00m 06s) |
[production] |
| 21:16 |
<andrew@deploy1001> |
Started deploy [horizon/deploy@b285acd]: update codfw1dev deploy from train-buster branch |
[production] |