|
2014-01-28
§
|
| 11:45 |
<springle> |
running fresh s5 dump for toolserver on db73 |
[production] |
| 10:41 |
<mutante> |
professor: revoked puppet cert, deleted salt minion key, powering down. bye bye |
[production] |
| 10:34 |
<mutante> |
disabled host and service checks/notifications for professor. running puppet on icinga |
[production] |
| 10:28 |
<mutante> |
removed professor from dsh,puppet,puppetmaster: on palladium Killing professor.pmtpa.wmnet...done. |
[production] |
| 04:19 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool db1040 for schema changes' |
[production] |
| 04:18 |
<springle> |
synchronized wmf-config/db-pmtpa.php 'depool db1040 for schema changes' |
[production] |
| 03:32 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-01-28 03:32:36+00:00 |
[production] |
| 03:11 |
<springle> |
synchronized wmf-config/db-pmtpa.php 'rotate s6 master, demote db1027, promote db1023' |
[production] |
| 03:11 |
<springle> |
synchronized wmf-config/db-eqiad.php 'rotate s6 master, demote db1027, promote db1023' |
[production] |
| 02:58 |
<springle> |
synchronized wmf-config/db-eqiad.php 'db1006 to LB 400. prep db1027 for s6 master rotation' |
[production] |
| 02:53 |
<springle> |
wikitech /a full again as per Ryan Lane email to ops@ on 2014-01-14. Deleted two oldest backup sets *2014012[01]*. |
[production] |
| 02:52 |
<LocalisationUpdate> |
completed (1.23wmf11) at 2014-01-28 02:52:02+00:00 |
[production] |
| 02:49 |
<LocalisationUpdate> |
completed (1.23wmf10) at 2014-01-28 02:25:44+00:00 |
[production] |
| 01:43 |
<^d> |
gerrit back up, running v2.8.1 stable now |
[production] |
| 01:36 |
<^d> |
gerrit down for upgrade |
[production] |
| 01:26 |
<bsitu> |
synchronized php-1.23wmf11/extensions/Flow 'Update Flow with some special contribs cherry-picks' |
[production] |
| 01:19 |
<ori> |
synchronized php-1.23wmf10/extensions/WikimediaEvents |
[production] |
| 01:18 |
<ori> |
synchronized php-1.23wmf11/extensions/WikimediaEvents |
[production] |
| 01:05 |
<ori> |
synchronized php-1.23wmf10/extensions/WikimediaEvents 'Ifc697cbe6: Revert I829790cd5, removing module storage logging' |
[production] |
| 00:57 |
<springle> |
synchronized wmf-config/db-eqiad.php 'repool db1006, warm up' |
[production] |
| 00:56 |
<ori> |
synchronized php-1.23wmf11/extensions/WikimediaEvents 'Ifc697cbe6: Revert I829790cd5, removing module storage logging' |
[production] |
|
2014-01-27
§
|
| 22:02 |
<csteipp> |
synchronized php-1.23wmf10/includes 'bug 60339' |
[production] |
| 21:58 |
<csteipp> |
synchronized php-1.23wmf11/includes 'bug 60339' |
[production] |
| 21:08 |
<csteipp> |
synchronized php-1.23wmf10/extensions/PdfHandler 'bug 60339' |
[production] |
| 21:07 |
<csteipp> |
synchronized php-1.23wmf11/extensions/PdfHandler 'bug 60339' |
[production] |
| 21:02 |
<ori> |
dns update |
[production] |
| 20:54 |
<ori> |
reload-vcl on cp1043: Backend host '"holmium.eikimrfis.org"' could not be resolved to an IP address: Name or service not known. (culprit: https://gerrit.wikimedia.org/r/#/c/109008/) |
[production] |
| 20:35 |
<aaron> |
synchronized php-1.23wmf11/includes/actions/InfoAction.php '5f94782c2bf09ce6c00b6ad37a68d08fef802fe5' |
[production] |
| 19:12 |
<reedy> |
synchronized docroot/bits/favicon/commons.ico |
[production] |
| 19:08 |
<reedy> |
synchronized wmf-config/ |
[production] |
| 19:07 |
<ori> |
synchronized php-1.23wmf10/extensions/WikimediaShopLink 'I21e34fe0f: Update WikimediaShopLink to master for PHP link insert' |
[production] |
| 19:05 |
<ori> |
synchronized php-1.23wmf11/extensions/WikimediaShopLink 'I21e34fe0f: Update WikimediaShopLink to master for PHP link insert' |
[production] |
| 18:45 |
<ori> |
cp1054 appears to have recovered, ops to investigate |
[production] |
| 18:37 |
<ori> |
dmesg on cp1054 [16167206.414548] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) |
[production] |
| 18:35 |
<ori> |
varnishd saturating cpu on cp1054 |
[production] |
| 17:26 |
<csteipp> |
synchronized php-1.23wmf10/extensions/TimedMediaHandler 'bug 56699 refix' |
[production] |
| 13:35 |
<springle> |
xtrabackup clone db1022 to db1006 |
[production] |
| 13:26 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool db1006' |
[production] |
| 10:44 |
<springle> |
synchronized wmf-config/db-eqiad.php 'repool db1015, warm up' |
[production] |
| 07:58 |
<springle> |
xtrabackup clone db1022 to db1015 |
[production] |
| 07:48 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool db1015' |
[production] |
| 07:00 |
<springle> |
synchronized wmf-config/db-eqiad.php 'repool db1023' |
[production] |
| 04:10 |
<springle> |
xtrabackup clone db1022 to db1023 |
[production] |
| 03:16 |
<springle> |
synchronized wmf-config/db-eqiad.php 'repool db1010, depool db1023' |
[production] |
| 03:09 |
<andrewbogott> |
I'm about to do a bunch of experimental bullshit in the production puppet repo, because I can't test network topology in labs. |
[production] |
| 02:44 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-01-27 02:44:23+00:00 |
[production] |
| 02:22 |
<LocalisationUpdate> |
completed (1.23wmf11) at 2014-01-27 02:22:27+00:00 |
[production] |
| 02:12 |
<LocalisationUpdate> |
completed (1.23wmf10) at 2014-01-27 02:12:03+00:00 |
[production] |