HP ProLiant MicroServer
出典: 2502.net
目次 |
インストール
2011-10-04
9月末にFreeBSD 9.0-BETA2にした。
Windows Home Serverでやってたファイルサーバ運用をこっちでやることにしたので、HDD増設したり。
- UFSなUSBメモリから起動してるけど、外付HDDがUSB接続なので、接続したまま起動しようとするとデバイス名が変わって悲惨。外付HDDを停止または外しておかないと / を見つけられなくて起動できない。ZFS rootにしちゃえば問題なさそうな気がするのでそのうちやる。
- WD25EZRXを4台買った。こいつはfirmwareがちゃんと4KiB sectorであることを報告してくれるらしいのだが、今回使ってるHDDケースのUSB接続だとsector sizeを取得できないみたいだったので、EARSと同様の対策が必要。dankogai先生の手法を参考にgnop作った。試した限り、かなり効果高い。
- USB 3.0のHUBつないだら動かんかった。ガッカリ。I/Fカードに直結なら問題なくいける。
- extとext2は一つのzpoolにしようと思ってたんだけど、vdev追加しようとしたら "mismatched replication levels" って怒られてやめた。-f付けると警告無視して作れるみたいだけど、非推奨らしいので。
- 同じraidz1同士でもディスクの数が違うと怒られるのには参った。そういうもんなんだなあ。
- 速度が遅いのは諦めた(外付のpoolに関してはsequentialでも100MB/sは絶対に出ないくらい。だいたい40-50MB/sくらい?)。ネットワーク越しに動画再生できりゃいいと割り切り。速度を諦めたのでCPUが非力なこのマシンでraidz2を使うことに。気にする場合はもっと速いCPUでSATAなりSASなりを使うべし。
> zpool status
pool: ext
state: ONLINE
scan: scrub in progress since Tue Oct 4 13:41:09 2011
29.8G scanned out of 15.8T at 54.3M/s, 84h42m to go
0 repaired, 0.18% done
config:
NAME STATE READ WRITE CKSUM
ext ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
da1.nop ONLINE 0 0 0
da2.nop ONLINE 0 0 0
da3.nop ONLINE 0 0 0
da4.nop ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
da5.nop ONLINE 0 0 0
da6.nop ONLINE 0 0 0
da7.nop ONLINE 0 0 0
da8.nop ONLINE 0 0 0
errors: No known data errors
pool: ext2
state: ONLINE
scan: scrub in progress since Tue Oct 4 13:41:19 2011
24.7G scanned out of 3.49T at 45.7M/s, 22h4m to go
0 repaired, 0.69% done
config:
NAME STATE READ WRITE CKSUM
ext2 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
da9.nop ONLINE 0 0 0
da10.nop ONLINE 0 0 0
da11.nop ONLINE 0 0 0
da12.nop ONLINE 0 0 0
da13.nop ONLINE 0 0 0
da14.nop ONLINE 0 0 0
da15.nop ONLINE 0 0 0
da16.nop ONLINE 0 0 0
da17.nop ONLINE 0 0 0
da18.nop ONLINE 0 0 0
errors: No known data errors
pool: tank
state: ONLINE
scan: scrub in progress since Tue Oct 4 13:41:22 2011
24.7G scanned out of 98.2G at 46.2M/s, 0h27m to go
0 repaired, 25.21% done
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0 ONLINE 0 0 0
ada1 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
ada2 ONLINE 0 0 0
ada3 ONLINE 0 0 0
errors: No known data errors
> zfs list
NAME USED AVAIL REFER MOUNTPOINT
ext 11.5T 173G 11.5T /ext
ext2 2.66T 10.9T 2.66T /ext2
tank 98.2G 3.47T 97.2G /tank
tank/ports 455M 3.47T 259M /usr/ports
tank/ports/distfiles 195M 3.47T 195M /usr/ports/distfiles
tank/ports/packages 31K 3.47T 31K /usr/ports/packages
tank/src 350M 3.47T 350M /usr/src
tank/var 102M 3.47T 212K /var
tank/var/crash 31K 3.47T 31K /var/crash
tank/var/db 101M 3.47T 101M /var/db
tank/var/db/pkg 262K 3.47T 262K /var/db/pkg
tank/var/empty 31K 3.47T 31K /var/empty
tank/var/log 386K 3.47T 386K /var/log
tank/var/mail 31K 3.47T 31K /var/mail
tank/var/run 54.5K 3.47T 54.5K /var/run
tank/var/tmp 32K 3.47T 32K /var/tmp
2011-05-04 OpenIndiana
OpenIndiana b148にOSを変えた。FreeBSDだと、どうもZFS使用時の安定性とトラブル発生時の挙動がよろしくない。そのうち良くなるとは思うけど…
DebianのAPTとIPSのコマンド対比表が無くてよー分からんかったので、ググって http://serverfault.com/questions/5947/linux-command-equivalents-in-opensolaris-and-vice-versa からかっぱらって来た。
APT IPS apt-get install pkg install apt-get remove pkg uninstall apt-cache search pkg search apt-get update pkg refresh (mostly redundant, pkg refreshes before installing) apt-get dist-upgrade pkg image-update editing sources.list pkg publisher / pkg set_publisher / pkg unset_publisher
2010-11-25 FreeBSD 8.1
iLOなカード(リモートアクセスカードキット)を買ったので、それ経由でUSBメモリにインストールした。インストール自体は通常のインストーラで、普通に入る。
ただし、今回使ったUSBメモリ(PQIのU273とかいうヤツ)の場合は /boot/loader.conf に kern.cam.boot_delay=10000 とか書いておかないとUSBメモリの認識前に / をmountしようとしてこけた。他のUSBメモリだとどうなるかは知らない。
あと、インストール後にiLOなカードの仮想CD-ROMドライブのマウント解除したらデバイスの番号が変わって /etc/fstab を s/da2/da0/g するハメになった。
FreeBSD 8.1 RELEASEのkernelだと、ネットワークに負荷かかった時(?)に bge0: watchdog timeout とか出てしばらくネットワークが死んでしまう現象が発生。色々いじったけど、最終的には FreeBSD 8.1 STABLE (2010-11-22頃l) のkernelに更新したら直った。
dmesgは下記のように。
Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.1-STABLE #0: Mon Nov 22 12:46:14 JST 2010
root@st01.2502.local:/usr/obj/usr/src/sys/GENERIC amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Athlon(tm) II Neo N36L Dual-Core Processor (1297.85-MHz K8-class CPU)
Origin = "AuthenticAMD" Id = 0x100f63 Family = 10 Model = 6 Stepping = 3
Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
Features2=0x802009<SSE3,MON,CX16,POPCNT>
AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!>
AMD Features2=0x8377f<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,Prefetch,OSVW,IBS,SKINIT,WDT,<b19>>
TSC: P-state invariant
real memory = 8589934592 (8192 MB)
avail memory = 8252555264 (7870 MB)
ACPI APIC Table: <HP ProLiant>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
ioapic0 <Version 2.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <HP ProLiant> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of fee00000, 1000 (3) failed
acpi0: reservation of ffb80000, 80000 (3) failed
acpi0: reservation of fec10000, 20 (3) failed
acpi0: reservation of fed80000, 1000 (3) failed
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, dff00000 (3) failed
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 4.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <PCI-PCI bridge> irq 16 at device 0.0 on pci1
pci2: <PCI bus> on pcib2
vgapci0: <VGA-compatible display> port 0xe800-0xe87f mem 0xfe000000-0xfe7fffff,0xfe8e0000-0xfe8fffff irq 16 at device 0.0 on pci2
pcib3: <ACPI PCI-PCI bridge> irq 18 at device 6.0 on pci0
pci3: <ACPI PCI bus> on pcib3
bge0: <HP NC107i PCIe Gigabit Server Adapter, ASIC rev. 0x5784100> mem 0xfe9f0000-0xfe9fffff irq 18 at device 0.0 on pci3
bge0: CHIP ID 0x05784100; ASIC REV 0x5784; CHIP REV 0x57841; PCI-E
miibus0: <MII bus> on bge0
brgphy0: <BCM5784 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge0: Ethernet address: d8:d3:85:af:77:c2
bge0: [FILTER]
atapci0: <ATI IXP700/800 SATA300 controller> port 0xd000-0xd007,0xc000-0xc003,0xb000-0xb007,0xa000-0xa003,0x9000-0x900f mem 0xfdfffc00-0xfdffffff irq 19 at device 17.0 on pci0
atapci0: [ITHREAD]
atapci0: AHCI v1.20 controller with 4 3Gbps ports, PM supported
ata2: <ATA channel 0> on atapci0
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci0
ata3: [ITHREAD]
ata4: <ATA channel 2> on atapci0
ata4: [ITHREAD]
ata5: <ATA channel 3> on atapci0
ata5: [ITHREAD]
ohci0: <OHCI (generic) USB controller> mem 0xfdffe000-0xfdffefff irq 18 at device 18.0 on pci0
ohci0: [ITHREAD]
usbus0: <OHCI (generic) USB controller> on ohci0
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfdfff800-0xfdfff8ff irq 17 at device 18.2 on pci0
ehci0: [ITHREAD]
usbus1: EHCI version 1.0
usbus1: <EHCI (generic) USB 2.0 controller> on ehci0
ohci1: <OHCI (generic) USB controller> mem 0xfdffd000-0xfdffdfff irq 18 at device 19.0 on pci0
ohci1: [ITHREAD]
usbus2: <OHCI (generic) USB controller> on ohci1
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xfdfff400-0xfdfff4ff irq 17 at device 19.2 on pci0
ehci1: [ITHREAD]
usbus3: EHCI version 1.0
usbus3: <EHCI (generic) USB 2.0 controller> on ehci1
pci0: <serial bus, SMBus> at device 20.0 (no driver attached)
atapci1: <ATI IXP700/800 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on pci0
ata0: <ATA channel 0> on atapci1
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci1
ata1: [ITHREAD]
isab0: <PCI-ISA bridge> at device 20.3 on pci0
isa0: <ISA bus> on isab0
pcib4: <ACPI PCI-PCI bridge> at device 20.4 on pci0
pci4: <ACPI PCI bus> on pcib4
ohci2: <OHCI (generic) USB controller> mem 0xfdffc000-0xfdffcfff irq 18 at device 22.0 on pci0
ohci2: [ITHREAD]
usbus4: <OHCI (generic) USB controller> on ohci2
ehci2: <EHCI (generic) USB 2.0 controller> mem 0xfdfff000-0xfdfff0ff irq 17 at device 22.2 on pci0
ehci2: [ITHREAD]
usbus5: EHCI version 1.0
usbus5: <EHCI (generic) USB 2.0 controller> on ehci2
acpi_button0: <Power Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
acpi_hpet1: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
device_attach: acpi_hpet1 attach returned 12
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff on isa0
atkbd: unable to set the command byte.
sc0: <System console> at flags 0x100 on isa0
sc0: CGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3d0-0x3db iomem 0xb8000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: unable to set the command byte.
ppc0: cannot reserve I/O port range
acpi_throttle0: <ACPI CPU Throttling> on cpu0
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 480Mbps High Speed USB v2.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 480Mbps High Speed USB v2.0
ad4: 1907729MB <WDC WD2001FASS-00W2B0 05.01D05> at ata2-master UDMA100 SATA 3Gb/s
ugen0.1: <ATI> at usbus0
uhub0: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <ATI> at usbus1
uhub1: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ugen2.1: <ATI> at usbus2
uhub2: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <ATI> at usbus3
uhub3: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen4.1: <ATI> at usbus4
uhub4: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen5.1: <ATI> at usbus5
uhub5: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
uhub4: 4 ports with 4 removable, self powered
uhub0: 5 ports with 5 removable, self powered
uhub2: 5 ports with 5 removable, self powered
ad6: 1907729MB <WDC WD2001FASS-00W2B0 05.01D05> at ata3-master UDMA100 SATA 3Gb/s
ad8: 1907729MB <WDC WD2001FASS-00W2B0 01.00101> at ata4-master UDMA100 SATA 3Gb/s
uhub5: 4 ports with 4 removable, self powered
ad10: 1907729MB <WDC WD2001FASS-00W2B0 05.01D05> at ata5-master UDMA100 SATA 3Gb/s
uhub1: 5 ports with 5 removable, self powered
uhub3: 5 ports with 5 removable, self powered
ugen3.2: <Avocent> at usbus3
ukbd0: <Keyboard> on usbus3
kbd2 at ukbd0
ums0: <Mouse> on usbus3
ums0: 3 buttons and [Z] coordinates ID=0
ums1: <Mouse REL> on usbus3
ums1: 3 buttons and [XYZ] coordinates ID=0
ugen3.3: <PQI> at usbus3
umass0: <PQI PQI USB Flash drive, class 0/0, rev 2.00/1.00, addr 3> on usbus3
umass0: SCSI over Bulk-Only; quirks = 0x0000
umass0: Get Max Lun not supported (USB_ERR_STALLED)
umass0:0:0:-1: Attached to scbus0
(probe0:umass-sim0:0:0:0): TEST UNIT READY. CDB: 0 0 0 0 0 0
(probe0:umass-sim0:0:0:0): CAM status: SCSI Status Error
(probe0:umass-sim0:0:0:0): SCSI status: Check Condition
(probe0:umass-sim0:0:0:0): SCSI sense: UNIT ATTENTION asc:28,0 (Not ready to ready change, medium may have changed)
da0 at umass-sim0 bus 0 scbus0 target 0 lun 0SMP: AP CPU #1 Launched!
da0:
<Generic USB Flash Disk 8.07> Removable Direct Access SCSI-2 device
da0: 40.000MB/s transfers
da0: 7538MB (15437824 512 byte sectors: 255H 63S/T 960C)
Trying to mount root from ufs:/dev/da0s1a
ZFS filesystem version 4
ZFS storage pool version 15
bge0: link state changed to UP
ZFS
2TB HDD (WD2001FASS) x 4 でRAIDZを構成。このHDDはヘッドのシーク音がステキに響く。個人的にはHDDがゴリゴリ言うのは嫌いじゃないので気にしない。
ZFS使う場合、Western DigitalのEARSを使うのはやめた方が良さそうだ。内部4Kbytes/sectorなのに、firmwareが512bytes/sectorと報告するせいで性能が出ないらしい。
メモリ周りのtuningをしてみたが、変わった気がしないのでdefaultに戻した。そのへんは既に十分賢くなっているらしい。
istgt
istgtを使ってiSCSI targetにして、Windows 7からCrystalDiskMarkを動かしてベンチマークを採ってみた。
Random R/W 4KB (QD=32) が異様に速いのはtarget側のcacheの関係か? テストサイズもっと大きくしてみるか。
TCP delayed ACKを無効にすると速いとかなんとか聞いたのでやってみたが、有意な差は見られなかった。
通常
-----------------------------------------------------------------------
CrystalDiskMark 3.0 x64 (C) 2007-2010 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]
Sequential Read : 99.779 MB/s
Sequential Write : 67.191 MB/s
Random Read 512KB : 91.935 MB/s
Random Write 512KB : 89.066 MB/s
Random Read 4KB (QD=1) : 5.592 MB/s [ 1365.3 IOPS]
Random Write 4KB (QD=1) : 5.632 MB/s [ 1375.0 IOPS]
Random Read 4KB (QD=32) : 49.457 MB/s [ 12074.6 IOPS]
Random Write 4KB (QD=32) : 42.634 MB/s [ 10408.7 IOPS]
Test : 1000 MB [I: 58.3% (291.2/499.9 GB)] (x5)
Date : 2010/11/25 21:27:45
OS : Windows 7 Ultimate Edition [6.1 Build 7600] (x64)
target側のTCP delayed ACK無効
net.inet.tcp.delayed_ack=0
-----------------------------------------------------------------------
CrystalDiskMark 3.0 x64 (C) 2007-2010 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]
Sequential Read : 98.810 MB/s
Sequential Write : 69.984 MB/s
Random Read 512KB : 89.489 MB/s
Random Write 512KB : 88.480 MB/s
Random Read 4KB (QD=1) : 5.043 MB/s [ 1231.2 IOPS]
Random Write 4KB (QD=1) : 5.334 MB/s [ 1302.3 IOPS]
Random Read 4KB (QD=32) : 49.232 MB/s [ 12019.4 IOPS]
Random Write 4KB (QD=32) : 33.507 MB/s [ 8180.4 IOPS]
Test : 1000 MB [I: 58.3% (291.2/499.9 GB)] (x5)
Date : 2010/11/25 21:42:25
OS : Windows 7 Ultimate Edition [6.1 Build 7600] (x64)
initiator側のTCP delayed ACK無効
Windows側 TcpAckFrequency=1
-----------------------------------------------------------------------
CrystalDiskMark 3.0 x64 (C) 2007-2010 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]
Sequential Read : 101.843 MB/s
Sequential Write : 67.027 MB/s
Random Read 512KB : 91.768 MB/s
Random Write 512KB : 82.987 MB/s
Random Read 4KB (QD=1) : 5.785 MB/s [ 1412.4 IOPS]
Random Write 4KB (QD=1) : 5.670 MB/s [ 1384.3 IOPS]
Random Read 4KB (QD=32) : 49.628 MB/s [ 12116.2 IOPS]
Random Write 4KB (QD=32) : 34.315 MB/s [ 8377.7 IOPS]
Test : 1000 MB [I: 58.3% (291.2/499.9 GB)] (x5)
Date : 2010/11/25 21:58:21
OS : Windows 7 Ultimate Edition [6.1 Build 7600] (x64)
Windows 7 側でstripeを作った場合
-----------------------------------------------------------------------
CrystalDiskMark 3.0 x64 (C) 2007-2010 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]
Sequential Read : 104.638 MB/s
Sequential Write : 65.890 MB/s
Random Read 512KB : 94.105 MB/s
Random Write 512KB : 91.106 MB/s
Random Read 4KB (QD=1) : 5.199 MB/s [ 1269.2 IOPS]
Random Write 4KB (QD=1) : 5.221 MB/s [ 1274.7 IOPS]
Random Read 4KB (QD=32) : 56.535 MB/s [ 13802.6 IOPS]
Random Write 4KB (QD=32) : 36.133 MB/s [ 8821.4 IOPS]
Test : 1000 MB [J: 0.0% (0.1/500.0 GB)] (x5)
Date : 2010/11/26 0:07:41
OS : Windows 7 Ultimate Edition [6.1 Build 7600] (x64)
