Huge kernel modules, SCSI is history

I have compiled the 2.6.25.4 kernel with many more modules than the kernel used in Puppy 3.01 -- it makes the live-CD ISO file 2MB bigger. On top of that I have put in some more applications and Abiword and Gnumeric are bigger.

Anyway, I used that handy graphical tool Gdmap (in the Filesystem menu) and looked around in /lib/modules/2.6.25.4. Nasty surprises!

The nastiest surprise of all is the size of the analog modem drivers. See the worst:
3.86MB drivers/char/Intel537.ko
1.37MB extra/agrmodem.ko
1.05MB drivers/char/Intel537.ko
1.1MB misc/pctel_hw.ko
445KB misc/esscom_hw.ko
547KB slmodem/slamr.ko
539KB ltmodem/ltmodem.ko

These modem drivers stand out as massively huge compared with most other drivers, and together they make up a huge chunk of the total space occupied in /lib/modules.

That Intel 537EP driver in particular -- I am tempted to take it out and make it a PET package only. How many people is that going to inconvenience I wonder?

Then there's the question of the SCSI drivers, that is, drivers for true-SCSI drives. The biggest of these is 293KB, but there are a lot of them, totalling 3.34MB.

Why are those true-SCSI drivers in Puppy anyway? I ask this as yesterday I had a telephone conversation with the Aussie guy who has anonymously donated some SCSI kit for me to experiment with. It was waitng for me at the Post Office on Friday and he phoned to make sure that I had received it.

Anyway, he made some interesting comments. He has had a lot of experience with SCSI, and he explained that the hard drives are becoming rare as they fail due to overheating. So apart from Sage and a few others hanging onto this old stuff, nobody out there is using it, and for anyone who wants to it is difficult to find a hard drive that still works.

He has confirmed what I suspected. Okay, I've got this SCSI gear now, so I'll have a go at getting it working, but apart from that I'm not interested. Perhaps I'll take out the entire collection of SCSI drivers from Puppy, for the 'standard' build. I'll make this a choice in Unleashed, so Puppy can still be built with SCSI drivers included -- so we can have a special build for those few who need it.

Posted on 19 May 2008, 17:20


Comments:

Posted on 19 May 2008, 20:12 by Gene Venable
Do People Still Use Modems?
I haven't touched an analog modem for years, since I was living in Russia in a building whose wiring didn't support DSL. I moved to ISBM, then DSL, then Cable, many years ago. The one thing that really startled me about Puppy has been all the modem support that comes with it. I think that modem support should be supplied, but how about conducting a survey to find out how many Puppy users even need it?


Posted on 19 May 2008, 21:48 by mouldy
dialup
I cant speak to other countries, but despite inroads of dsl and cable, dialup is still a major factor in USA, at least 40 to 50% of the market and in rural areas you have to either be wealthy or need it for your job to justify satellite. Dialup and satellite are the ONLY choices. And if you really want to downlload big files like movies, etc, you have to signup for the "buisiness" satellite service at much higher price or you easily outstrip your bandwidth limitations. Just doing the official updates to windows systems might overdo you bandwith, LOL...

I am even noticing on other linux forums people requesting built in soft modem support like Puppy has. Puppy friendliness is getting noticed.

I have been looking at the French Slitaz linux. It has PPP kernel support and thats pretty much all. They did finally add some marginal script dialer thingy that I couldnt get to work. I somehow dont think they even tried it out (maybe none of them has access to dialup?), just stuck it in there after complaints. I copied wvdial 1.41 binary (77kb) from Puppy and manually created necessary files it looks for. Worked like a charm. They had however not included pl2303.ko nor ch341.ko so couldnt use usb adapter with my modem. I dont think people, especially in Europe, appreciate the need for dialup support. A distribution called Beatrix even came out couple years or so ago without PPP compiled in. Austrumi may have started without it also.


Posted on 19 May 2008, 22:23 by coolpup
re Dial-up modems

The answer is yes, people still do use dial-up modems. The more appropriate question would be, Do people still need them?

According to this web site the global broadband penetration rate is less than 5%:
http://www.internetworldstats.com/dsl.htm

Someone appears helpless without one:
http://www.helpscreen.com.au/index.php?msgid=1328231008&cid=2

How do you expect Puppy Linux to achieve world domination without its built-in modem support?


Posted on 19 May 2008, 22:47 by kirk
modems
Generally I'm in favor of hardware support over applications when it comes to cutting size. But when you think about the chance that Puppy will not work with any given software modem is maybe 80%, taking away one 3.8MB modem module would probably only increase that number by a couple percent.

As far a SCSI goes, it's very rare in PCs, which is what Puppy seems to target. I still see them in some high end servers I work on, but even there they are being replaced by SATA. SCSI is used because of it speed and hardware RAID capabilities, though the speed advantage is nearly gone. No one manufactures PCs with SCSI because the costs are too high.





Posted on 19 May 2008, 23:43 by nfisher
Modem support
Up until recently (this year) modem support was extremely important to me and the only way I could connect when away from home. I know these modules are sometimes huge but I'd rather have them right in the live cd.


Posted on 19 May 2008, 23:48 by prehistoric2
special case modules
This kind of issue keeps cropping up. I've been bothered by all the junk in RAM I never use. My pragmatic solution has been to have enough RAM to get started and create a large swap partition, so anything which doesn't get used will be swapped out.

This is inelegant. What I keep thinking we need is a special cases sfs which is only mounted as needed. With a list of special cases modules and a script which copies those still loaded at shutdown to your pup_save.2fs (if they are not already present,) we would have a situation where people who don't use them pay a minimal price for this overhead and people who do need them have them available.


Posted on 19 May 2008, 24:30 by pizzasgood
zdrv_xxx.sfs
What you described is basically what the zdrv_xxx.sfs file is.


Posted on 20 May 2008, 2:15 by dogone
Hardware support selection after first boot
Easing out support for legacy hardware will always pose a challenge so a consistent strategy is a worthwhile goal.

A good compromise that can be updated with every Puppy release is use of one or more supplemental sfs boot time modules. You initially boot Puppy without it/them, determine what additional hardware support you require, enable one or more and reboot. Puppy boots with only what you need thereafter.

Puppy 4.1 could move SCSI support and other than basic modem support to such a module. Puppy 4.8 could drop SCSI support from the module to make room for other legacy hardware, etc.


Posted on 20 May 2008, 2:36 by dogone
follow-up
Given the above, some may be concerned that Puppy won't always "just work" at first boot. Well, it will work at second boot and given Puppy's mission, this seems a reasonable compromise. We have only to compare Puppy's performance to distros six times it's size that don't always "just work".

Being small isn't easy.


Posted on 20 May 2008, 3:06 by Ted Dog
LZMA
French Slitaz linux (also in English) uses LZMA to compress the kernel and the intrd.gz it is a small patch and saves about 1M on the kernel and 20-30M on the initrd (which contains about 50M if gzipped, 90M raw) Puppy could save about 2M in the way it uses the kernel & initrd.
Based on puppys' small size target using LZMA on the two boot files can give about 2M reduction, That is not the same as the LZMA squashfile we tried. about a year ago


Posted on 20 May 2008, 3:56 by DonT
Analog Modem Drivers
I would like to second what mouldy has posted.

A surprisingly large number of people in the USA (me included) still have only dial-up service available to them.

And, people new to Linux are prone to abandon their efforts with a distribution when they cannot connect to the internet. With it's modem probe and available drivers, Puppy is, without doubt, the friendliest Linux available to the dial-up user. I know how impressed I was to find that two of my computers with softmodems would connect readily "right out of the box" using Puppy. (Didn't have to use a serial modem as always before!)

Excluding a heavy modem driver may be required to meet the size parameters of Puppy. And, as kirk has posted, it would only affect those percentages with the modem. But, along with this, there is the potential for lost converts.

As a suggestion, if the heavier drivers are left out and made into pet packages, then would it be possible to put a bit of dialog into the modem wizard that, after a successful probe for the excluded modem, would announce that the driver was available and where? Perhaps, the affected person would then make the effort to get the modem working.

Dogone posted, "Being small isn't easy." That's a great phrase to explain Puppy.






Posted on 20 May 2008, 6:29 by jeffrey
SCSI & modem support
I don't use SCSI disk on PCs but use it exclusively on IBM p-series machines. I've never experienced or heard of the overheating argument. SCSI disks were used because of reliability and speed - they were the best available, and I have no problem locating old ones to expand my existing platforms. As kirk points out they are expensive, which probably restricted their use in the PC market. I have about 50 drives, most of which have been operational for 5 years.
I think that the built-in modem support is great. I use a dial-up connection on a PC with external modem (because when I set it up no Linux would support my card out of the box). My laptop winmodem was useless with Linux but I see that Puppy 4 will dial with it (although I haven't got wvdial to establish a connection yet, but it looks like it's down to modem initialisation options now). Well done Barry - keep the modem drivers, slowly drop SCSI support if only one or two complain.
BTW it looks like you've listed drivers/char/Intel537.ko twice in your post for some reason.


Posted on 20 May 2008, 10:03 by BarryK
536EP
Ah yes, that second one was supposed to be the Intel536.ko module.


Posted on 20 May 2008, 11:07 by kirk
lzma
Using lzma on the initrd. That's an interesting idea. looks like Busybox supports lzma too. Maybe just use lzma on the modules?




Posted on 20 May 2008, 11:22 by BarryK
RE: lzma
Ted Dog, yes that is interesting. I'm not in the mood to do a kernel patch for 2.6.25.4 right now, as it took me a couple of days compiling and recompiling the kernel and 3rd-party modules ...and I don't want to go through it all again.
But certainly I'll keep it in mind for the future.


Posted on 20 May 2008, 13:21 by Ted Dog
LZMA & new
The Slitaz linux team just came out with a new alpha, still under 25M, very clean looking. Try it out if you have a chance. I notice LZMA compresses kernel objects very well when grouped together, lots of copied code I suppose. If the modules are placed into initrd and deleted from RAM after applied, and before squashfile is loaded, the RAM would be better used than Slitaz currently does.


Posted on 20 May 2008, 15:52 by ANOSage
Linux Days - Geneva, tomorrow
Ted Dog only related the technical content about the new Slitaz, but their piece draws attention to this event:
http://www.linuxdays.ch/
No doubt that Puppy should be represented. Anyone want to send out an olive branch to the much maligned marksouth asking him to attend on our behalf?! Anyone else in that region.

In response to the DUN modem discussion, at last we have the Achilles' Heal. Punish the Yanks for installing the idiot son who has destroyed the world. Deprive them of all the DUN apps - remove the lotfrom Puppy! [and consider installing a guide to humour in its place].


Posted on 20 May 2008, 22:08 by Ted Dog
LZMA & new
Sage cut it out, I remember Crawford TX as good place to get unlimited sweet Tea while cramming for exams. The Cafe no longer does that, why should we be punished more, the lurking reporters caused that sweet Tea loss, not Bush!
I never meet Bush in Waco, I did shake hands with Al Gore Jr. at the Waco airport
(still washing that hand continuously )



Posted on 20 May 2008, 23:42 by kirk
lzma-busybox
Barry,

Assuming you've put all the modules in the initrd, you could use lzma to compress the modules. Then recompile BusyBox with lzma enabled so you would have lzmacat to decompress the modules. I tried it with a couple of modules. It's about 20% smaller than with gzip. The kernel patch wouldn't save that much anyway.

I posted a binary for lzma here:

http://www.murga-linux.com/puppy/viewtopic.php?p=199205#199205

Busybox only uncompresses.



Posted on 21 May 2008, 3:03 by kirk
more lzma tests
Played with lzma some more. The uncompressed kernel modules for 2.6.25 is 38MB. If you compress each module with gzip, the total is 15MB. If you compress each module with lzma, the total is 12MB. So I don't know if it's worth it for 3MB compressed space. Lzma is a little slower than gzip too. Though this new version seems quite fast.




Posted on 21 May 2008, 4:16 by Ted Dog
LZMA thread moved
I'm glad someone is thinking along the same lines
http://www.murga-linux.com/puppy/viewtopic.php?p=199205#199205
I've replied with additional ideas, and code frag.


Posted on 21 May 2008, 4:57 by Ted Dog
Loss of unlimited sweet tea
Here is a picture of the place that used to give unlimited sweet tea, and its Aussie connection.

http://www.crawfordtx.com/photos_howard.htm

note it was never that crowded when I studied there.


Posted on 21 May 2008, 12:17 by crafty-one
modems
Is there a way to 'mount' an ".sfs" file to Puppy WITHOUT having to reboot..?

Also - can an already mounted ".sfs" file be 'unmounted' WITHOUT having to reboot..?

If so for either - then maybe the 'initrd' could have some script(s) to do this 'dynamic' mount / unmount during the "boot up" stage - so as to give access to drivers etc. that may be needed - then free up the used RAM space for use by the 'booted' Puppy..!

crafty.
.



Posted on 22 May 2008, 24:37 by pizzasgood
mounting
It's possible to do a normal mount / unmount easily.
mount my_stuff.sfs /mnt/data -o loop
umount /mnt/data

This is not the same as putting it into the UnionFS tree, which would merge the file with the rest of the filesystem (like happens when you do the normal rebooting method).

IIRC, adding a layer to the tree dynamically is theoretically possible but unstable. I don't know much on the subject though.


Posted on 23 May 2008, 21:44 by Aitch
SCSI
Barry
I'm rather disappointed to see your comments about SCSI booting support,(Especially after all the hassle trying to get support kit to you, where others have failed) which to me goes hand in hand with dual/quad processor support, as I mess about with old server hardware, which being built for speed & reliability uses both, & I totally dispute the notion that SCSI drives fail due to overheating
Server cases have better airflow/cooling systems than Desktop PCs & I personally have had 5 H/D IDE failures but no SCSI failures. Many have lifetime warranty & will outlast most PCs
As for PCI SCSI card support, there are 2 main cards Adaptec 2940UW & Tekram DC-395UW which are in abundant use and easy to support, which would enable old server hardware, currently going very cheap, to be run on Puppy [Even better with multi-processor support
I am currently using SCSI, but just can't boot from it & I'd hate to lose that ability
Thanks, Aitch


Posted on 24 May 2008, 9:50 by BarryK
Re: SCSI support
Aitch,
I won't be removing support for SCSI, at least not for sometime. I only said that I might leave the SCSI modules out of the main official live-CD, but build another slightly bigger ISO with the modules in it. So there would be a choice. I didn't state anything other than that. Even that is a 'maybe' as I might leave them in the main official live-cd.

One of the purposes of this blog is to 'ruminate', think things over, rethink previous ideas, propose new ideas. What ends up in a final release is another matter.


Posted on 24 May 2008, 10:52 by wildpossum
SCSI is a tiny minority
I should first say that I work with SCSI equipped servers and disks, so I do know about SCSI and how to support them. However I would never use SCSI on a home machine unless I had a convenient basement. They are too noisy and eat too much power.

However I feel that anyone messing around with SCSI is probably capable of adding such support to the Puppy images themselves.

The irony is that those who need the modem modules which are far larger are more like to need driver support out of the box.

Such are the growing pains of Puppy.


Posted on 26 May 2008, 9:20 by Aitch
SCSI booting support
@wildpossom
"However I feel that anyone messing around with SCSI is probably capable of adding such support to the Puppy images themselves".
You may work with SCSI servers, but that doesn't make you right in this sweeping statement, as I AM such a messer with old servers, & don't have a clue what is required as I've never worked on/with computers at all, & if I had your knowledge & experience, I would probably come onto the forums & demonstrate exactly what you profess to be expert on, to save Barry the hassle, and to help resolve a booting problem that appears to have remained unresolved for nearly 3 years, although apparently early kernels had SCSI booting support built-in, & I've read that it's just a matter of re-enabling a couple of modules for it all to come back to life
Are we not all part of the same pack, running, jumping & frolicking together? :)
eh, possum?

Aitch


Posted on 26 May 2008, 9:33 by Aitch
SCSI
a quick PS

Thanks for the clarification, & update, Barry.
I await your further consideration, and hope not too many modems dominate the efforts, as, on the forums, at least, Wifi far outways modems, in terms of calls for help.
Booting, grub, in particular, is by far the biggest overall noob-killer, and brings even experienced linux-heads to near despair, trying to resolve 'broken windoze' calls
Broken documentation links is the other.
....just my 2 pence [Ha Ha, UK] worth of feedback.

Aitch


Posted on 26 May 2008, 15:44 by wildpossum
loading SCSI modules
Well not having seen your post, I don't know what you have or haven't tried. But nowadays pretty much all Linux distros have SCSI drivers as modules, and you have to load the right one for the controller before disk access is attempted. It's not reasonable to require the init scripts to probe every possible SCSI driver and slow down the 99% of users who don't have SCSI. So this is something you have to bite the bullet and learn at some point anyway.

Why don't you paste a URL to your forum post, and I'll try to comment on what you might have to do.