My PC at work died the other day and while scrounging together hardware to keep me running until we order new i5 based PCs for the techs I had the joy of fighting the elusive heisenbug of the Vostro 200.

Long story short: if you have issues with a PCI-E card like video cutting out randomly and never returning it’s because the designers were on meth. The solution is to only have RAM in DIMM slots 3 and 4. If you put anything in 1 or 2 you’ll want to commit suicide.

What the hell, Dell. This is why I build my PCs.

I seem to run into this more often than I would like. The Dell MD3000i (a rebadged IBM which is also available as a generic LSI product) is a horrible SAN. The worst part is that you can have terrible performance with no way to know why unless you run these cli commands to figure out that the write cache is disabled. There’s no SNMP and no other reliable way to monitor the damn things, which is probably because they want you to buy their Equilogic line.

Anyway, I had a battery die on a controller again and write cache was disabled. Replacing the battery on that controller did not fix it — it was still saying the write cache was suspended. After a LOT of poking it finally came back, but I don’t know exactly how or why. I currently have a theory though that I hope helps someone out some day.

1) Known issue that MD3000i units ship with writeCacheEnabled=true and mirrorCacheEnabled=true, which puts the write cache into a suspended mode because it can’t mirror with only one controller. (They come with one controller unless you order two)

2) Fixing the above problem requires disabling both, then re-enabling only the write cache.

3) Trying the above did not work. Mirror disabled, but still suspended. Somehow with mirror disabled and forcing it to ignore the battery (cacheWithoutBatteriesEnabled=true) would get write cache working again. That’s not a good or safe solution, though.

4) Disabling all write cache settings, doing an offline/online of each controller, and then enabling all the write caches and fixing paths seemed to work.

I think it refuses to mirror the cache which keeps it suspended because the controllers still have some weird cache setting that doesn’t get cleared until you offline/online both controllers to “reset” them. It’s currently the best theory I have, but doesn’t explain why I could force it to work by setting cacheWithoutBatteriesEnabled=true.

tl;dr this SAN is a big box of shit.

Today was a hellish day. Late in the afternoon when we’re supposed to be winding down and heading out one of our shared webhosting servers started freaking out. No changes to any software or settings — nothing to suspect — but mysql-proxy kept crashing. We need this utility because we have a lot of old customers with configs from a long time ago when someone thought it was OK to have the database on the webserver.

We’ve never had much luck with mysql-proxy as we’ve seen it crash its fair amount of times or not start properly on boot, but this was unending. Core dumps weren’t giving anything useful; logs were no better. Here’s the MySQL-Proxy alternative that we should have implemented ages ago. Note, we’re running FreeBSD, so adapt to your own OS/Linux distro:

What you need:


net/socat
net/haproxy


FreeBSD’s rc.conf:


# mysql_proxy alternative
socat_enable="YES"
socat_flags="UNIX-LISTEN:/tmp/mysql.sock,fork,reuseaddr,unlink-early,mode=777 TCP:127.0.0.1:3306"
haproxy_enable="YES"


HAProxy’s config:


global
log 127.0.0.1 local0
maxconn 4096
daemon
#debug
#quiet
#
defaults
log global
mode tcp
option tcplog
option dontlognull
option tcp-smart-accept
option tcp-smart-connect
retries 3
maxconn 200
#
listen mysql :3306
mode tcp
option mysql-check
balance roundrobin
server mysql1 1.2.3.4:3306 check port 3306


Ultimately, HAProxy manages the connections very well and socat fixes any old clients trying to talk over the /tmp/mysql.sock file.

Enjoy.