|
2015-12-07
§
|
| 17:00 |
<hashar> |
Restarted Jenkins to start the ZeroMQ publisher https://phabricator.wikimedia.org/T120668 |
[production] |
| 16:56 |
<_joe_> |
upgrading pybal on primaries in esams, ulsfo, codfw |
[production] |
| 16:44 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration for en.wiktionary [[gerrit:254476]] and Enable subpages on custom aliases from 112 to 119 [[gerrit:254475]] (duration: 00m 27s) |
[production] |
| 16:43 |
<_joe_> |
installing pybal on eqiad backups |
[production] |
| 16:43 |
<jynus> |
restarting mysql, upgrading and rebooting mysql on es1019 |
[production] |
| 16:36 |
<_joe_> |
installing the new pybal version on the backup lvss in esams and ulsfo |
[production] |
| 16:31 |
<_joe_> |
upgrading pybal on all the backup LVSs in codfw |
[production] |
| 16:28 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Rights configuration on fa.wikipedia [[gerrit:254486]] (duration: 00m 29s) |
[production] |
| 16:28 |
<_joe_> |
upgrading pybal on lvs1007-12 |
[production] |
| 16:19 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration for ur.wikipedia [[gerrit:254665]] (duration: 00m 30s) |
[production] |
| 16:18 |
<_joe_> |
uploaded pybal 1.13.2 to reprepro |
[production] |
| 16:14 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Translate project namesapce for jbowiki [[gerrit:255954]] (duration: 00m 28s) |
[production] |
| 15:39 |
<moritzm> |
uploaded openssl 1.0.2e-1~wmf1 to carbon |
[production] |
| 15:37 |
<yurik> |
deployed kartotherian |
[production] |
| 15:25 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool es1019; es1017 at 100% load; pool es1015 with low weight (duration: 00m 28s) |
[production] |
| 14:56 |
<yurik> |
deployed latest tilerator |
[production] |
| 14:46 |
<_joe_> |
also restarted pybal on lvs3003 |
[production] |
| 14:42 |
<_joe_> |
restarted pybal on lvs1006 |
[production] |
| 14:38 |
<hashar> |
restarting Jenkins |
[production] |
| 14:32 |
<hashar> |
Jenkins lost a bunch of executors :/ |
[production] |
| 14:30 |
<hashar> |
CI / Zuul stalled somehow |
[production] |
| 12:56 |
<jynus> |
rolling restart, configuration upgrade of es1015 |
[production] |
| 12:20 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool es1015; es1013 at 100% load; pool es1017 with low weight (duration: 00m 28s) |
[production] |
| 11:08 |
<YuviPanda> |
restarting pdns on holmium |
[production] |
| 10:52 |
<jynus> |
database and system maintenance to es1017 |
[production] |
| 10:43 |
<hashar> |
CI / zuul / nodepool recovered. Root cause was some malfunction in openstack wmflabs |
[production] |
| 10:20 |
<YuviPanda> |
restarted nova-conductor and scheduler on labcontrol1001 |
[production] |
| 10:07 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool es1013 (lower weight for now) and depool es1017 (duration: 00m 41s) |
[production] |
| 10:05 |
<hashar> |
stopped Nodepool. Can not create instances anymore on wmflabs ( https://phabricator.wikimedia.org/T120586 ) |
[production] |
| 09:46 |
<hashar> |
restarting Nodepool on labnodepool1001.eqiad.wment |
[production] |
| 09:40 |
<hashar> |
CI / Zuul stalled. Nodepool can no more spawn instances :-/ |
[production] |
| 09:27 |
<godog> |
nodetool decommission restbase1008 |
[production] |
| 09:13 |
<jynus> |
es1013 maintenance (mysql restart, upgrade, possible reboot) |
[production] |
| 08:27 |
<_joe_> |
uploaded etcd 2.2 package from stretch to jessie-wikimedia |
[production] |
| 03:56 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Mon Dec 7 03:56:49 UTC 2015 (duration 1h 32m 22s) |
[production] |
| 02:24 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.7) (duration: 09m 59s) |
[production] |
|
2015-12-05
§
|
| 18:30 |
<gwicke> |
started nodetool decommission on restbase1008 |
[production] |
| 11:35 |
<reedy@tin> |
Synchronized wmf-config/CommonSettings.php: Disable common password password policy to come in wmf.8 (duration: 00m 28s) |
[production] |
| 11:23 |
<reedy@tin> |
Purged l10n cache for 1.27.0-wmf.5 |
[production] |
| 11:22 |
<reedy@tin> |
Synchronized php-1.27.0-wmf.7/extensions/WikimediaMaintenance/refreshMessageBlobs.php: Less waiting for slaves (duration: 00m 28s) |
[production] |
| 11:13 |
<reedy@tin> |
Synchronized docroot and w: Add jobqueue-labs to noc (duration: 00m 28s) |
[production] |
| 08:59 |
<bblack> |
offlined db1019 megacli disk 32:11 |
[production] |
| 06:09 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sat Dec 5 06:09:07 UTC 2015 (duration 3h 44m 18s) |
[production] |
| 02:24 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.7) (duration: 09m 59s) |
[production] |