The magical command to get it to work (I did this after running vmware-toolbox, I don't know if that's an essential step):
sudo dpkg-reconfigure xserver.xorg
Trick found here.
Sunday, February 03, 2008
Saturday, January 12, 2008
pf miscellanea on FreeBSD
pf Status
Install and run /usr/ports/sysutils/pftop. Press "5" and "8" to see how your rules and queues are doing. "?" for help on what else you can do.
/usr/ports/net/trafshow is also very useful for seeing what going through your network cards.
pf Logging
To get logging to work on FreeBSD, make a rule like:
(pf.conf) pass in log quick on $ext_inf
then enable the pf logging interface with:
$ kldload ~/test_pf/pflog.ko
$ ifconfig pflog0 up
then you can watch the logged stuff with:
$ tcpdump -n -e -ttt -i pflog0
queues of queues
If you try to have multilayer queues, with borrowing, things won't work as you expect, e.g.:
queue bandwidth 20kb q1(a,b) bandwidth 20kb borrow
queue bandwidth 10kb a borrow
queue bandwidth 10kb b borrow
queue bandwidth 20kb q2(y,z) bandwidth 20kb borrow
queue bandwidth 10kb y borrow
queue bandwidth 10kb z borrow
If queue a fills up, it seems to borrow from b, but then when a+b reaches 20kbit, it doesn't seem to borrow (at all / very well) from q2, even if q2 is idle.
Install and run /usr/ports/sysutils/pftop. Press "5" and "8" to see how your rules and queues are doing. "?" for help on what else you can do.
/usr/ports/net/trafshow is also very useful for seeing what going through your network cards.
pf Logging
To get logging to work on FreeBSD, make a rule like:
(pf.conf) pass in log quick on $ext_inf
then enable the pf logging interface with:
$ kldload ~/test_pf/pflog.ko
$ ifconfig pflog0 up
then you can watch the logged stuff with:
$ tcpdump -n -e -ttt -i pflog0
queues of queues
If you try to have multilayer queues, with borrowing, things won't work as you expect, e.g.:
queue bandwidth 20kb q1(a,b) bandwidth 20kb borrow
queue bandwidth 10kb a borrow
queue bandwidth 10kb b borrow
queue bandwidth 20kb q2(y,z) bandwidth 20kb borrow
queue bandwidth 10kb y borrow
queue bandwidth 10kb z borrow
If queue a fills up, it seems to borrow from b, but then when a+b reaches 20kbit, it doesn't seem to borrow (at all / very well) from q2, even if q2 is idle.
FreeBSD pf traffic shaping rules
Handy link: http://www.openbsd.org/faq/pf/queueing.html
My pf.conf does this:
- My outgoing is 256kbit, this uses 240kbit to leave a bit of overhead so my cable modem doesn't end up doing shaping as well.
- Some redirection of traffic to internal clients
- All outgoing web (port 80) traffic is transparently sent to squid
- No security.. No inbound ports are actively blocked.
- Brute force poking and prodding of my ssh port will get you put in jail for the night.
- Install /usr/ports/security/expiretable, and put into rc.conf:
expiretable_enable="yes"
expiretable_flags="-d -t 1d bruteforce"
- Prioritisation:
- Priority is given to VoIP traffic (identified by port, and originating device on my LAN)
- SSH is next
- Then everything else
- Then Bittorrent (identified by setting my BT client to make all traffic come from a given port)
- Bittorrent is limited to about 15Kbytes/sec outgoing, as there is no borrowing enabled on the "bt" queue
"Ah-ha" moments:
- You create the queues on the outgoing (internet) interface
- You can (if you want) say which queue stuff is go to into as it comes in on the LAN side. Then, if it ends up going out to the internet, it'll end up in the designated queue.
The rules...
ext_if="sk0"
int_if="re0"
internal_net="192.168.60.0/24"
cablemodem="192.168.100.1"
voipbox="192.168.60.5"
# BT client set to have all outbound traffic as being from port 43123
# ports 43000:43999 forwarded to an internal machine
voipports="{5060,6000:6005}"
# Options: tune the behavior of pf, default values are given.
#set timeout { interval 10, frag 30 }
#set timeout { tcp.first 120, tcp.opening 30, tcp.established 86400 }
#set timeout { tcp.closing 900, tcp.finwait 45, tcp.closed 90 }
#set timeout { udp.first 60, udp.single 30, udp.multiple 60 }
#set timeout { icmp.first 20, icmp.error 10 }
#set timeout { other.first 60, other.single 30, other.multiple 60 }
#set timeout { adaptive.start 0, adaptive.end 0 }
#set limit { states 10000, frags 5000 }
#set loginterface none
#set optimization normal
#set block-policy drop
#set require-order yes
#set fingerprints "/etc/pf.os"
martians = "{ 127.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, \
10.0.0.0/8, 169.254.0.0/16, 192.0.2.0/24, \
0.0.0.0/8, 240.0.0.0/4 }"
table persist
# Normalization: reassemble fragments and resolve or reduce traffic ambiguities.
scrub in all
altq on $ext_if cbq bandwidth 240Kb queue { voip, tcpack, ssh_int, ssh_bulk, def, bt,btack }
queue bt bandwidth 128Kb priority 1 cbq( red ecn)
queue btack bandwidth 6Kb priority 2 cbq(borrow red)
queue def bandwidth 6Kb priority 3 cbq(borrow red default)
queue ssh_bulk bandwidth 6Kb priority 3 cbq(borrow red)
queue ssh_int bandwidth 6Kb priority 4 cbq(borrow red)
queue voip bandwidth 80Kb priority 6 cbq(borrow red)
queue tcpack bandwidth 8Kb priority 7 cbq(borrow red)
nat on $ext_if from $internal_net to any -> ($ext_if)
rdr on $ext_if proto {tcp,udp} from any to any port 43000:43999 -> 192.168.60.42
rdr on $ext_if proto {tcp,udp} from any to any port 25777 -> 192.168.60.42
# rdp = 3389
rdr on $ext_if proto {tcp,udp} from any to any port 3389 -> 192.168.60.42
rdr on $ext_if proto tcp from any to any port 5900 -> 192.168.60.42
#SIP/VoIP
rdr on $ext_if proto {tcp,udp} from any to any port $voipports -> $voipbox
# squid
rdr on $int_if proto tcp from any to any port http -> 127.0.0.1 port 3128
antispoof for $ext_if
antispoof for $int_if
block quick from
pass in quick on $ext_if from $cablemodem to any
pass out quick on $ext_if from any to $cablemodem
block drop in quick on $ext_if from $martians to any
block drop out quick on $ext_if from any to $martians
pass in on $ext_if inet proto tcp from any to any port 22 \
flags S/SA keep state \
(max-src-conn 15, max-src-conn-rate 5/3, \
overload flush global)
################################################################
##
## VOIP
pass in quick on $int_if inet proto {tcp,udp} from any to any port $voipports keep state queue (voip,tcpack)
pass in quick on $int_if inet proto {tcp,udp} from any port $voipports to any keep state queue (voip,tcpack)
pass out quick on $ext_if inet proto {tcp,udp} from any port $voipports to any keep state queue (voip,tcpack)
pass out quick on $ext_if inet proto {tcp,udp} from any to any port $voipports keep state queue (voip,tcpack)
pass in quick on $int_if from $voipbox to any keep state queue (voip,tcpack)
################################################################
##
## BT
pass in quick on $int_if inet proto {tcp,udp} from any port 43123 to any keep state queue (bt,btack)
pass in quick on $int_if inet proto {tcp,udp} from any to any port 43123 keep state queue (bt,btack)
pass in quick on $int_if inet proto {tcp,udp} from any port 43123 to any keep state queue (bt,btack)
pass in quick on $int_if inet proto {tcp,udp} from any to any port 6881:6889 keep state queue (bt,btack)
################################################################
##
## SSH
pass in quick on $int_if inet proto tcp from any to any port 22 \
keep state flags S/SA queue(ssh_int, ssh_bulk)
################################################################
pass in quick on $int_if inet proto tcp all modulate state flags S/SA queue(def, tcpack)
pass in quick on $int_if inet proto { udp, icmp, gre } all keep state
# security schmecurity
pass in all
pass out all
My pf.conf does this:
- My outgoing is 256kbit, this uses 240kbit to leave a bit of overhead so my cable modem doesn't end up doing shaping as well.
- Some redirection of traffic to internal clients
- All outgoing web (port 80) traffic is transparently sent to squid
- No security.. No inbound ports are actively blocked.
- Brute force poking and prodding of my ssh port will get you put in jail for the night.
- Install /usr/ports/security/expiretable, and put into rc.conf:
expiretable_enable="yes"
expiretable_flags="-d -t 1d bruteforce"
- Prioritisation:
- Priority is given to VoIP traffic (identified by port, and originating device on my LAN)
- SSH is next
- Then everything else
- Then Bittorrent (identified by setting my BT client to make all traffic come from a given port)
- Bittorrent is limited to about 15Kbytes/sec outgoing, as there is no borrowing enabled on the "bt" queue
"Ah-ha" moments:
- You create the queues on the outgoing (internet) interface
- You can (if you want) say which queue stuff is go to into as it comes in on the LAN side. Then, if it ends up going out to the internet, it'll end up in the designated queue.
The rules...
ext_if="sk0"
int_if="re0"
internal_net="192.168.60.0/24"
cablemodem="192.168.100.1"
voipbox="192.168.60.5"
# BT client set to have all outbound traffic as being from port 43123
# ports 43000:43999 forwarded to an internal machine
voipports="{5060,6000:6005}"
# Options: tune the behavior of pf, default values are given.
#set timeout { interval 10, frag 30 }
#set timeout { tcp.first 120, tcp.opening 30, tcp.established 86400 }
#set timeout { tcp.closing 900, tcp.finwait 45, tcp.closed 90 }
#set timeout { udp.first 60, udp.single 30, udp.multiple 60 }
#set timeout { icmp.first 20, icmp.error 10 }
#set timeout { other.first 60, other.single 30, other.multiple 60 }
#set timeout { adaptive.start 0, adaptive.end 0 }
#set limit { states 10000, frags 5000 }
#set loginterface none
#set optimization normal
#set block-policy drop
#set require-order yes
#set fingerprints "/etc/pf.os"
martians = "{ 127.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, \
10.0.0.0/8, 169.254.0.0/16, 192.0.2.0/24, \
0.0.0.0/8, 240.0.0.0/4 }"
table
# Normalization: reassemble fragments and resolve or reduce traffic ambiguities.
scrub in all
altq on $ext_if cbq bandwidth 240Kb queue { voip, tcpack, ssh_int, ssh_bulk, def, bt,btack }
queue bt bandwidth 128Kb priority 1 cbq( red ecn)
queue btack bandwidth 6Kb priority 2 cbq(borrow red)
queue def bandwidth 6Kb priority 3 cbq(borrow red default)
queue ssh_bulk bandwidth 6Kb priority 3 cbq(borrow red)
queue ssh_int bandwidth 6Kb priority 4 cbq(borrow red)
queue voip bandwidth 80Kb priority 6 cbq(borrow red)
queue tcpack bandwidth 8Kb priority 7 cbq(borrow red)
nat on $ext_if from $internal_net to any -> ($ext_if)
rdr on $ext_if proto {tcp,udp} from any to any port 43000:43999 -> 192.168.60.42
rdr on $ext_if proto {tcp,udp} from any to any port 25777 -> 192.168.60.42
# rdp = 3389
rdr on $ext_if proto {tcp,udp} from any to any port 3389 -> 192.168.60.42
rdr on $ext_if proto tcp from any to any port 5900 -> 192.168.60.42
#SIP/VoIP
rdr on $ext_if proto {tcp,udp} from any to any port $voipports -> $voipbox
# squid
rdr on $int_if proto tcp from any to any port http -> 127.0.0.1 port 3128
antispoof for $ext_if
antispoof for $int_if
block quick from
pass in quick on $ext_if from $cablemodem to any
pass out quick on $ext_if from any to $cablemodem
block drop in quick on $ext_if from $martians to any
block drop out quick on $ext_if from any to $martians
pass in on $ext_if inet proto tcp from any to any port 22 \
flags S/SA keep state \
(max-src-conn 15, max-src-conn-rate 5/3, \
overload
################################################################
##
## VOIP
pass in quick on $int_if inet proto {tcp,udp} from any to any port $voipports keep state queue (voip,tcpack)
pass in quick on $int_if inet proto {tcp,udp} from any port $voipports to any keep state queue (voip,tcpack)
pass out quick on $ext_if inet proto {tcp,udp} from any port $voipports to any keep state queue (voip,tcpack)
pass out quick on $ext_if inet proto {tcp,udp} from any to any port $voipports keep state queue (voip,tcpack)
pass in quick on $int_if from $voipbox to any keep state queue (voip,tcpack)
################################################################
##
## BT
pass in quick on $int_if inet proto {tcp,udp} from any port 43123 to any keep state queue (bt,btack)
pass in quick on $int_if inet proto {tcp,udp} from any to any port 43123 keep state queue (bt,btack)
pass in quick on $int_if inet proto {tcp,udp} from any port 43123 to any keep state queue (bt,btack)
pass in quick on $int_if inet proto {tcp,udp} from any to any port 6881:6889 keep state queue (bt,btack)
################################################################
##
## SSH
pass in quick on $int_if inet proto tcp from any to any port 22 \
keep state flags S/SA queue(ssh_int, ssh_bulk)
################################################################
pass in quick on $int_if inet proto tcp all modulate state flags S/SA queue(def, tcpack)
pass in quick on $int_if inet proto { udp, icmp, gre } all keep state
# security schmecurity
pass in all
pass out all
Thursday, March 29, 2007
gmirror raid1 on boot drive
Setting up gmirror on the boot drive, on an already running system:
http://people.freebsd.org/~rse/mirror/
http://people.freebsd.org/~rse/mirror/
Thursday, October 26, 2006
FreeBSD : Postfix (SMTP + SSL + TLS) + Courier (IMAP + POP3 + SSL) + Cyrus SASL, with virtual delivery
Links
* http://www.postfix.org/SASL_README.html
* http://www.unixpeople.com/HOWTO/postfix.solaris.html
* http://www.pacnog.org/pacnog1/day1/mail/courier.html
Postfix
/usr/local/etc/postfix/master.cf
# enables SMTP over SSL on port 465. Note the command line arguments for smtpd.
smtps inet n - n - - smtpd -o smtpd_tls_wrappermode=yes -o smtpd_sasl_auth_enable=yes
/usr/local/etc/postfix/main.cf
queue_directory = /var/spool/postfix
command_directory = /usr/local/sbin
daemon_directory = /usr/local/libexec/postfix
mail_owner = postfix
unknown_local_recipient_reject_code = 550
mynetworks = 192.168.1.0/24, 127.0.0.0/8
smtpd_banner = $myhostname ESMTP
debug_peer_level = 2
debugger_command =
PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin
xxgdb $daemon_directory/$process_name $process_id & sleep 5
sendmail_path = /usr/local/sbin/sendmail
newaliases_path = /usr/local/bin/newaliases
mailq_path = /usr/local/bin/mailq
setgid_group = maildrop
html_directory = no
manpage_directory = /usr/local/man
sample_directory = /usr/local/etc/postfix
readme_directory = no
transport_maps = hash:/usr/local/etc/postfix/transport
smtpd_sender_restrictions = reject_unknown_sender_domain, reject_non_fqdn_sender
smtpd_recipient_restrictions = permit_mynetworks, reject_unauth_destination, reject_non_fqdn_recipient, reject_unknown_recipient_domain
virtual_mailbox_domains = sun-research.com
virtual_mailbox_base = /mirror/mail/
virtual_mailbox_maps = hash:/usr/local/etc/postfix/vmailbox
virtual_minimum_uid = 100
virtual_uid_maps = static:125
virtual_gid_maps = static:125
virtual_alias_maps = hash:/usr/local/etc/postfix/virtual
smtpd_sasl_security_options = noanonymous
smtpd_use_tls = yes
smtpd_tls_cert_file = /usr/local/etc/postfix/postfix.pem
smtpd_tls_key_file = $smtpd_tls_cert_file
smtpd_tls_auth_only = yes
broken_sasl_auth_clients = yes
smtpd_sasl_auth_enable = yes
smtpd_helo_required = yes
Don't forget to make your PEM file!
/usr/local/etc/postfix/transport (Do a "postconf transport" after editing)
# null entry means to handle this domain ourselves
company.com :
.company.com :
# enable next line to use a smart host for all outgoing mail
# * smtp:smtp.my.isp.com
/usr/local/etc/postfix/vmailbox ("postconf vmailbox")
This file sets up the users that postfix will receive mail for, and where to store their mail.
# Don't forget the trailing / for maildir format!
user1@company.com company.com/user1/
user2@company.com company.com/user2/
Using virtual_mailbox_base directory from main.cf above:
mkdir -p $virtual_mailbox_base
cd $virtual_mailbox_base
maildirmake user1
maildirmake user2
chown -R postfix:postfix .
Courier
/usr/local/etc/courier-imap/imapd
IMAPDSTART=YES
/usr/local/etc/courier-imap/pop3d
POP3DSTART=YES
/usr/local/etc/courier-imap/imapd-ssl
IMAPDSTARTTLS=YES
/usr/local/etc/courier-imap/pop3d-ssl
POP3_STARTTLS=YES
Make your TLS certificates:
* Edit /usr/local/etc/courier-imap/{pop3d,imapd}.cnf and put in your details.
* Put CN = mail.companyname.com, or whatever your clients use to connect.
* Then:
cd /usr/local/share/courier-imap/
./mkpop3dcert
./mkimapdcert
/etc/rc.conf
courier_authdaemond_enable="YES"
courier_imap_pop3d_enable="YES"
courier_imap_imapd_enable="YES"
courier_imap_pop3d_ssl_enable="YES"
courier_imap_imapd_ssl_enable="YES"
saslauthd_enable="YES"
saslauthd_flags="-r -a rimap -O localhost"
/usr/local/etc/authlib/authdaemonrc
authmodulelist="authuserdb"
Edit /usr/local/etc/authlib/userdb
This controls who can log in to get their email, and who is allowed to relay SMTP.
# All one line...
user1@company.com
uid=125|
gid=125|
home=/mirror/mail/company.com/user1|
mail=/mirror/mail/company.com/user1|
systempw=|
Run "makeuserdb" after editing the userdb file!
Configure Cyrus SASL by creating/editing /usr/local/lib/sasl2/smtpd.conf with these two lines:
pwcheck_method: saslauthd
mech_list: PLAIN LOGIN
Remember to open firewall ports!
And I think that's it. :)
* http://www.postfix.org/SASL_README.html
* http://www.unixpeople.com/HOWTO/postfix.solaris.html
* http://www.pacnog.org/pacnog1/day1/mail/courier.html
Postfix
/usr/local/etc/postfix/master.cf
# enables SMTP over SSL on port 465. Note the command line arguments for smtpd.
smtps inet n - n - - smtpd -o smtpd_tls_wrappermode=yes -o smtpd_sasl_auth_enable=yes
/usr/local/etc/postfix/main.cf
queue_directory = /var/spool/postfix
command_directory = /usr/local/sbin
daemon_directory = /usr/local/libexec/postfix
mail_owner = postfix
unknown_local_recipient_reject_code = 550
mynetworks = 192.168.1.0/24, 127.0.0.0/8
smtpd_banner = $myhostname ESMTP
debug_peer_level = 2
debugger_command =
PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin
xxgdb $daemon_directory/$process_name $process_id & sleep 5
sendmail_path = /usr/local/sbin/sendmail
newaliases_path = /usr/local/bin/newaliases
mailq_path = /usr/local/bin/mailq
setgid_group = maildrop
html_directory = no
manpage_directory = /usr/local/man
sample_directory = /usr/local/etc/postfix
readme_directory = no
transport_maps = hash:/usr/local/etc/postfix/transport
smtpd_sender_restrictions = reject_unknown_sender_domain, reject_non_fqdn_sender
smtpd_recipient_restrictions = permit_mynetworks, reject_unauth_destination, reject_non_fqdn_recipient, reject_unknown_recipient_domain
virtual_mailbox_domains = sun-research.com
virtual_mailbox_base = /mirror/mail/
virtual_mailbox_maps = hash:/usr/local/etc/postfix/vmailbox
virtual_minimum_uid = 100
virtual_uid_maps = static:125
virtual_gid_maps = static:125
virtual_alias_maps = hash:/usr/local/etc/postfix/virtual
smtpd_sasl_security_options = noanonymous
smtpd_use_tls = yes
smtpd_tls_cert_file = /usr/local/etc/postfix/postfix.pem
smtpd_tls_key_file = $smtpd_tls_cert_file
smtpd_tls_auth_only = yes
broken_sasl_auth_clients = yes
smtpd_sasl_auth_enable = yes
smtpd_helo_required = yes
Don't forget to make your PEM file!
/usr/local/etc/postfix/transport (Do a "postconf transport" after editing)
# null entry means to handle this domain ourselves
company.com :
.company.com :
# enable next line to use a smart host for all outgoing mail
# * smtp:smtp.my.isp.com
/usr/local/etc/postfix/vmailbox ("postconf vmailbox")
This file sets up the users that postfix will receive mail for, and where to store their mail.
# Don't forget the trailing / for maildir format!
user1@company.com company.com/user1/
user2@company.com company.com/user2/
Using virtual_mailbox_base directory from main.cf above:
mkdir -p $virtual_mailbox_base
cd $virtual_mailbox_base
maildirmake user1
maildirmake user2
chown -R postfix:postfix .
Courier
/usr/local/etc/courier-imap/imapd
IMAPDSTART=YES
/usr/local/etc/courier-imap/pop3d
POP3DSTART=YES
/usr/local/etc/courier-imap/imapd-ssl
IMAPDSTARTTLS=YES
/usr/local/etc/courier-imap/pop3d-ssl
POP3_STARTTLS=YES
Make your TLS certificates:
* Edit /usr/local/etc/courier-imap/{pop3d,imapd}.cnf and put in your details.
* Put CN = mail.companyname.com, or whatever your clients use to connect.
* Then:
cd /usr/local/share/courier-imap/
./mkpop3dcert
./mkimapdcert
/etc/rc.conf
courier_authdaemond_enable="YES"
courier_imap_pop3d_enable="YES"
courier_imap_imapd_enable="YES"
courier_imap_pop3d_ssl_enable="YES"
courier_imap_imapd_ssl_enable="YES"
saslauthd_enable="YES"
saslauthd_flags="-r -a rimap -O localhost"
The saslauthd_flags set it to use the full realm (-r), use remote IMAP server for authenticating against (-a rimap) and to contact the IMAP server on this machine (-O localhost). The default is to use pam that will do things like checking against passwd.
/usr/local/etc/authlib/authdaemonrc
authmodulelist="authuserdb"
Edit /usr/local/etc/authlib/userdb
This controls who can log in to get their email, and who is allowed to relay SMTP.
# All one line...
user1@company.com
uid=125|
gid=125|
home=/mirror/mail/company.com/user1|
mail=/mirror/mail/company.com/user1|
systempw=|
Run "makeuserdb" after editing the userdb file!
Configure Cyrus SASL by creating/editing /usr/local/lib/sasl2/smtpd.conf with these two lines:
pwcheck_method: saslauthd
mech_list: PLAIN LOGIN
Remember to open firewall ports!
And I think that's it. :)
Monday, April 10, 2006
Rebuilding a vinum RAID5 array when a drive has failed
A drive in your vinum RAID5 array has failed...
%> vinum list
D drive6 State: down /dev/twed6s1a A: 114325/114470 MB (99%)
P raid5volume.p0 R5 State: degraded Subdisks: 8 Size: 1020 MB
S raid5volume.p0.s6 State: crashed D: drive6 Size: 145 MB
Go find a replacement drive. Once you have a replacement drive in hand:
* Shutdown PC (my FreeBSD + twe driver get confused with a hot-plug of a drive)
* Remove faulty drive and replace with a good one
* Restart PC
%> vinum list
D drive6 State: referenced unknown A: 0/0 MB
P raid5volume.p0 R5 State: degraded Subdisks: 8 Size: 1020 MB
S raid5volume.p0.s6 State: stale D: drive6 Size: 145 MB
When it says "referenced", this seems to means either:
* Partition (fdisk) the new drive. If you see it already has a partition on it, be careful, and double-check you're working on the right drive!
* Disklabel the new drive
%> vinum printconfig
drive drive6 device unknown
Vinum doesn't know which phsyical drive/device 'drive6' refers to.
* Remove the referenced drive from vinum:
%> vinum rm drive6
The referenced drive6 disappears from vinum's list
%> vinum list
S raid5volume.p0.s6 State: stale D: Size: 145 MB
* Create a text file, e.g. foo.txt, with the single line:
drive drive6 device /dev/twed6s1a
* Load the text configuration file into vinum
%> vinum config foo.txt
%> vinum list
D drive6 State: up /dev/twed6s1a A: 114325/114470 MB (99%)
P raid5volume.p0 R5 State: degraded Subdisks: 8 Size: 1020 MB
S raid5volume.p0.s6 State: stale D: drive6 Size: 145 MB
* Start the plex
%> vinum start raid5volume.p0
Reviving raid5volume.p0.s6 in the background
vinum[870]: reviving raid5volume.p0.s6
* Wait patiently
You can run "vinum list" to see the percentage it is through the rebuild. Eventually, you'll see:
vinum[870]: raid5volume.p0.s6 is up
* Done!
Your array is back.
%> vinum list
V raid5volume State: up Plexes: 1 Size: 1020 MB
P raid5volume.p0 R5 State: up Subdisks: 8 Size: 1020 MB
S raid5volume.p0.s6 State: up D: drive6 Size: 145 MB
%> vinum checkparity -f -v raid5volume.p0
Checking at 1020 MB (99%) raid5volume.p0 has correct parity
%> vinum list
D drive6 State: down /dev/twed6s1a A: 114325/114470 MB (99%)
P raid5volume.p0 R5 State: degraded Subdisks: 8 Size: 1020 MB
S raid5volume.p0.s6 State: crashed D: drive6 Size: 145 MB
Go find a replacement drive. Once you have a replacement drive in hand:
* Shutdown PC (my FreeBSD + twe driver get confused with a hot-plug of a drive)
* Remove faulty drive and replace with a good one
* Restart PC
%> vinum list
D drive6 State: referenced unknown A: 0/0 MB
P raid5volume.p0 R5 State: degraded Subdisks: 8 Size: 1020 MB
S raid5volume.p0.s6 State: stale D: drive6 Size: 145 MB
When it says "referenced", this seems to means either:
- You forgot to plug the drive power/data cable in
- Clues to this is that at least one /dev/twedXs1a is missing. Check your /var/log/messages, the drives that vinum has in its list, and what there is in /dev.
- This drive is completely blank (to be expected, if you replaced the drive)
* Partition (fdisk) the new drive. If you see it already has a partition on it, be careful, and double-check you're working on the right drive!
* Disklabel the new drive
%> vinum printconfig
drive drive6 device unknown
Vinum doesn't know which phsyical drive/device 'drive6' refers to.
* Remove the referenced drive from vinum:
%> vinum rm drive6
The referenced drive6 disappears from vinum's list
%> vinum list
S raid5volume.p0.s6 State: stale D: Size: 145 MB
* Create a text file, e.g. foo.txt, with the single line:
drive drive6 device /dev/twed6s1a
* Load the text configuration file into vinum
%> vinum config foo.txt
%> vinum list
D drive6 State: up /dev/twed6s1a A: 114325/114470 MB (99%)
P raid5volume.p0 R5 State: degraded Subdisks: 8 Size: 1020 MB
S raid5volume.p0.s6 State: stale D: drive6 Size: 145 MB
* Start the plex
%> vinum start raid5volume.p0
Reviving raid5volume.p0.s6 in the background
vinum[870]: reviving raid5volume.p0.s6
* Wait patiently
You can run "vinum list" to see the percentage it is through the rebuild. Eventually, you'll see:
vinum[870]: raid5volume.p0.s6 is up
* Done!
Your array is back.
%> vinum list
V raid5volume State: up Plexes: 1 Size: 1020 MB
P raid5volume.p0 R5 State: up Subdisks: 8 Size: 1020 MB
S raid5volume.p0.s6 State: up D: drive6 Size: 145 MB
%> vinum checkparity -f -v raid5volume.p0
Checking at 1020 MB (99%) raid5volume.p0 has correct parity
Creating a vinum RAID5 volume
8 * 120GB hard drives, on a 3Ware controller (/dev/twe)
----------vinum.conf----------
volume raid5volume
drive drive0 device /dev/twed0s1a
drive drive1 device /dev/twed1s1a
drive drive2 device /dev/twed2s1a
drive drive3 device /dev/twed3s1a
drive drive4 device /dev/twed4s1a
drive drive5 device /dev/twed5s1a
drive drive6 device /dev/twed6s1a
drive drive7 device /dev/twed7s1a
plex org raid5 491k
sd length 114470m drive drive0
sd length 114470m drive drive1
sd length 114470m drive drive2
sd length 114470m drive drive3
sd length 114470m drive drive4
sd length 114470m drive drive5
sd length 114470m drive drive6
sd length 114470m drive drive7
------------------------------
vinum resetconfig
vinum create vinum.conf
vinum init raid5volume.p0
(...wait five and a half hours...)
newfs /dev/vinum/raid5volume
mount /dev/vinum/raid5volume /raid
Wednesday, April 05, 2006
Cheeseburger beats grapes
At the place where I get my lunch they also sell fruit, etc. This kid (probably 6-7 years old) was choosing something. His mother says "Pick one piece of fruit", so the kid grabs a bunch of grapes. The mother (slightly flustered/rushed) says "No! One piece!". The kid puts back grapes and grabs a peach. The mother gets impatient "No! Don't bother. Put it back ...". She takes the peach from the kid, and puts it back on the display. "I'll get you a cheeseburger at Macquarie."
Surprisingly, neither the mother or any of her 3 kids (all 4-7 age range) were fat.
Surprisingly, neither the mother or any of her 3 kids (all 4-7 age range) were fat.
Tuesday, April 04, 2006
Subscribe to:
Posts (Atom)