|
2013-12-23
§
|
| 22:15 |
<awight> |
updated crm from ce73eaa0ea6019f76c63d0cca097bf828eb933e2 to 87dea8afb964d0622f3ee92b6e120915846fde56 |
[production] |
| 20:06 |
<paravoid> |
cr1-eqiad/cr1-sdtpa: new BGP policy to reset MED for XO |
[production] |
| 16:59 |
<cmjohnson1> |
dns update |
[production] |
| 15:30 |
<cmjohnson1> |
labnet1001 relocating racks C4 to B3 eqiad |
[production] |
| 13:24 |
<springle_> |
starting xtrabackup clone db1001 to db1016 |
[production] |
| 10:39 |
<akosiaris> |
uploaded php5_5.3.10-1ubuntu3.9+wmf1 to apt.wikimedia.org |
[production] |
| 09:54 |
<paravoid> |
rebooting cp1065 with Linux 3.2 |
[production] |
| 07:43 |
<springle> |
taking down db1016 for xfs_check, locked up and: XFS (dm-0): xlog_space_left: head behind tail |
[production] |
| 05:57 |
<springle> |
synchronized wmf-config/db-eqiad.php 'warm up db1033 in s1' |
[production] |
| 02:26 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Dec 23 02:26:35 UTC 2013 |
[production] |
| 02:13 |
<LocalisationUpdate> |
completed (1.23wmf8) at Mon Dec 23 02:13:09 UTC 2013 |
[production] |
| 02:07 |
<LocalisationUpdate> |
completed (1.23wmf7) at Mon Dec 23 02:07:23 UTC 2013 |
[production] |
| 00:50 |
<springle> |
xtrabackup clone db1049 to db1033 |
[production] |
| 00:37 |
<springle> |
synchronized wmf-config/db-eqiad.php 'rebalance group loads while db1050 out' |
[production] |
|
2013-12-22
§
|
| 19:35 |
<apergos> |
db1050 boot: (during fsck) hung at "The disk drive for /a is not ready yet or not present. Continue waiting, S to skip.." etc |
[production] |
| 19:34 |
<apergos> |
db1050 boot: "[ 65.625950] device-mapper: table: 252:3: snapshot: Snapshot cow pairing for exception table handover failed" |
[production] |
| 19:33 |
<ori> |
synchronized wmf-config/db-eqiad.php 'Depool db1050' |
[production] |
| 19:32 |
<ori> |
updated /a/common to {{Gerrit|If33894260}}: Depool db1050; unresponsive |
[production] |
| 19:27 |
<apergos> |
powrcycling db1050, inaccessible |
[production] |
| 02:28 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sun Dec 22 02:28:45 UTC 2013 |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.23wmf8) at Sun Dec 22 02:15:27 UTC 2013 |
[production] |
| 02:08 |
<LocalisationUpdate> |
completed (1.23wmf7) at Sun Dec 22 02:08:39 UTC 2013 |
[production] |
|
2013-12-20
§
|
| 21:37 |
<mutante> |
cp1065 - currently init 1 sitting at maintenance but not powered down, i will poke later or anyone can |
[production] |
| 20:44 |
<mutante> |
cp1065 - was 3.11.0-13-generic, rebooting for 3.2.0-57-generic |
[production] |
| 20:38 |
<mutante> |
powercycled cp1065 again, will upgrade kernel this time |
[production] |
| 19:09 |
<mutante> |
cp1065 - [19060.904886] BUG: scheduling while atomic: kworker/11:0/27073/0x00000200 |
[production] |
| 19:09 |
<mutante> |
powercycling cp1065 |
[production] |
| 16:18 |
<andrewbogott> |
merged a couple of tiny OpenStackManager patches on wikitech |
[production] |
| 14:34 |
<hashar> |
Jenkins removed http_proxy env variables from lanthanum.eqiad.wmnet does not play well with git urls :D |
[production] |
| 14:29 |
<hashar> |
Jenkins setting http_proxy and https_proxy on lanthanum.eqiad.wmnet to point to carbon.wikimedia.org (eqiad web proxy) |
[production] |
| 13:52 |
<paravoid> |
powercycling cp1065 |
[production] |
| 11:53 |
<akosiaris> |
added akosiaris account on all junipers. Also added faidon account wherever it did not exist and synchronized its uid (only one uid change on csw2-esams happened) |
[production] |
| 11:16 |
<hashar> |
updated Jenkins job linting operations/apache-config.git , now detects rewrites.conf not being updated when redirects.dat has been changed. |
[production] |
| 10:19 |
<apergos> |
powercycled cp1065 and had to manually start varnish, same as yesterday |
[production] |
| 02:04 |
<springle> |
synchronized wmf-config/db-eqiad.php 'vslow query load balancing' |
[production] |