|
2013-11-27
§
|
| 18:12 |
<paravoid> |
kill -9 gdb on cp3012, attached to varnish frontend |
[production] |
| 11:28 |
<ori-l> |
faidon switched gdash.wm.o from professor.pmtpa -> tungsten.eqiad behind misc-varnish & rebooted ssl1 in tampa |
[production] |
| 11:11 |
<apergos> |
ssl1 rebooted itself about 15 mins ago, no idea why |
[production] |
| 10:20 |
<ariel> |
synchronized wmf-config/db-eqiad.php 'db1019 (s3) back to full weight in the pool' |
[production] |
| 10:19 |
<ariel> |
updated /a/common to {{Gerrit|If5ebd6194}}: db1019 (s3) back to full weight in pool |
[production] |
| 10:08 |
<apergos> |
shot some old puppet processes hogging memory on db9 (from march and earlier) |
[production] |
| 09:49 |
<apergos> |
there was no mount /srv/pagecounts on labstore4, so rsync to /exp/pagecounts wrote to and filled /; did the mkdir and now things seem ok |
[production] |
| 08:00 |
<ariel> |
synchronized wmf-config/db-eqiad.php 'warm up db1019 (s3) aftr lvm resize' |
[production] |
| 07:59 |
<ariel> |
updated /a/common to {{Gerrit|I50354e622}}: warm up db1019 (s3) after lvm resize |
[production] |
| 07:38 |
<apergos> |
rebooting db1019 after kernel upgrade, fix for broken xfs_growfs |
[production] |
| 07:02 |
<ariel> |
synchronized wmf-config/db-eqiad.php 'depool db1019 (s3) temporarily for lvm resize' |
[production] |
| 07:01 |
<ariel> |
updated /a/common to {{Gerrit|I4372bb602}}: depool db1019 (s3) temporarily for lvm resize |
[production] |
| 02:40 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Nov 27 02:40:52 UTC 2013 |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.23wmf5) at Wed Nov 27 02:15:19 UTC 2013 |
[production] |
| 02:08 |
<LocalisationUpdate> |
completed (1.23wmf4) at Wed Nov 27 02:08:28 UTC 2013 |
[production] |
| 00:32 |
<springle> |
stopping replication on sanitarium db1054:3308 and labsdb1002:3308 while restoring dewiki to labs |
[production] |
|
2013-11-26
§
|
| 22:58 |
<mutante> |
deploying gerrit 92925 & 91124 (apache-config), makes /entity/ URLs on wikidata 303 and removes non-existent noncom wiki |
[production] |
| 21:15 |
<mutante> |
praseodymium, cerium, xenon: disabled icinga notifications and scheduled 1yr downtime for host and all services per gwicke, they are test hosts and not prod. |
[production] |
| 21:04 |
<bblack> |
killed attached gdb on cp3012's varnishd frontend, which restarted it... |
[production] |
| 20:21 |
<csteipp> |
synchronized php-1.23wmf4/extensions/TimedMediaHandler 'bug56699' |
[production] |
| 20:16 |
<csteipp> |
synchronized php-1.23wmf5/extensions/TimedMediaHandler 'bug56699' |
[production] |
| 20:05 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Not wmf5 day today |
[production] |
| 19:33 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Revert to 1.23wmf4 on wikisources |
[production] |
| 19:23 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: All non wikipedias to 1.23wmf5 |
[production] |
| 19:21 |
<MaxSem> |
Reindexing GeoData |
[production] |
| 19:16 |
<reedy> |
updated /a/common to {{Gerrit|Idb0e7956e}}: slaves to full steam |
[production] |
| 08:00 |
<ori-l> |
re-pooled ssl1003 |
[production] |
| 07:57 |
<ori-l> |
depooling ssl1003 for quick test of puppet config |
[production] |
| 06:22 |
<apergos> |
pwercycled labstore1001, unreachable, nothing on mgmt console |
[production] |
| 06:00 |
<springle> |
synchronized wmf-config/db-eqiad.php 'slaves to full steam after package upgrade' |
[production] |
| 05:59 |
<paravoid> |
applying workaround for Ganglia XSS https://github.com/ganglia/ganglia-web/issues/218 |
[production] |
| 05:32 |
<springle> |
synchronized wmf-config/db-eqiad.php 'warm up slaves after package upgrade' |
[production] |
| 04:51 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool slaves for package upgrade' |
[production] |
| 04:05 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool slaves for package upgrade' |
[production] |
| 03:29 |
<Tim> |
deploying varnish normalize_path (I687f6773) |
[production] |
| 03:14 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Tue Nov 26 03:14:08 UTC 2013 |
[production] |
| 02:25 |
<Tim> |
deploying new generated redirects.conf |
[production] |
| 02:24 |
<LocalisationUpdate> |
completed (1.23wmf5) at Tue Nov 26 02:24:12 UTC 2013 |
[production] |
| 02:16 |
<LocalisationUpdate> |
completed (1.23wmf4) at Tue Nov 26 02:16:11 UTC 2013 |
[production] |
| 02:03 |
<awight> |
updated crm from 9979fe9a0f456499ff5be261aa41ca601a88dc13 to 82af6188f5e019ab23636e78719e17c4cb838859 |
[production] |
| 01:59 |
<awight> |
rollback crm from 3e7f0d57023e98e6be0e1d7539b2abd04b72ad99 to 9979fe9a0f456499ff5be261aa41ca601a88dc13 |
[production] |
| 01:58 |
<awight> |
updated crm from 9979fe9a0f456499ff5be261aa41ca601a88dc13 to 3e7f0d57023e98e6be0e1d7539b2abd04b72ad99 |
[production] |
| 01:57 |
<springle> |
synchronized wmf-config/db-eqiad.php 'slaves to full steam' |
[production] |
| 01:22 |
<springle> |
synchronized wmf-config/db-eqiad.php 'warm up slaves after package upgrade' |
[production] |
| 00:43 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool slaves for packages upgrade' |
[production] |